| About Oodle on Nintendo Switch |
|
|
Oodle for Switch is provided as a lib.
lib/liboo2coreswitch.a
lib/liboo2extswitch.a
The debug build of the Oodle lib is also provided. Generally the release build of Oodle should be linked
with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems.
OodleX file IO on Switch uses stdio.
OodleX Malloc on Switch uses stdlib malloc.
NOTE to run the examples on Switch, you must mount some directory for them to work on,
and change the paths used to be relative to that mount.
You could add :
then run "example_lz_chart c:test_file"
In example_lz : Example demonstrating LZ compression and decompression these would need to be changed to be relative to the mount point with an
absolute path :
static const char * in_name = "oodle_example_input_file";
static const char * out_name = "oodle_example_output_file";
Discussion
Initialize Oodle with no threads and minimal systemsParameters
Return Value
Discussion
You must call OodleX_Init or OodleX_Init_NoThreads before any other Oodle function that you expect to work.
Pair with OodleX_Shutdown_NoThreads.
This function does not enable the Oodle IOQ or WorkMgr. No async jobs or IO will work.
All memory->memory compressors will work.
Pair with OodleX_Shutdown_NoThreads
Discussion
Free a pointer allocated with OodleXMalloc_IOAligned Parameters
Discussion
returns the alignment of OodleXMallocBig pointers Return Value
Discussion
Should be >= OODLEX_IO_MAX_ALIGNMENT
Discussion
Start a thread running threadfuncDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
enum OodleXHandleDeleteIfDone
{
OodleXHandleDeleteIfDone_No = 0,
OodleXHandleDeleteIfDone_Yes = 1,
OodleXHandleDeleteIfDone_Force32 = 0x40000000
};
Discussion
Pass OodleXHandleDeleteIfDone_Yes to handle status checks to delete the handle if it's done.
This is the main way to free an OodleXHandle
Enumerants
| OodleXMalloc_ValidatePointer |
|
|
Discussion
debug check if a pointer is a valid mallocParameters
| ptr | pointer to validate
|
| bytes | size of allocation if known; -1 if not
|
Return Value
| return | true if the malloc headers are all okay
|
Discussion
Should work on OodleXMalloc and OodleXMallocBig pointers.
Bytes can be -1 if unknown, but there will be less validation checks.
ValidatePointer is most useful if the OodleXMalloc debug thunk layer is installed in OodleX_Init.
| Frequently Asked Questions |
|
|
Oodle FAQ
Frequently Asked Questions :
| OODLE_JOB_MAX_DEPENDENCIES |
|
|
Discussion
Maximum number of dependencies Oodle will ever pass to a RunJob callback
| t_fp_OodleCore_Plugin_RunJob |
|
|
Discussion
Function pointer type for OodleCore_Plugins_SetJobSystemParameters
| dependencies | array of handles of other pending jobs. All guaranteed to be nonzero.
|
| num_dependencies | number of dependencies. Guaranteed to be no more than OODLE_JOB_MAX_DEPENDENCIES.
|
| user_ptr | is passed through from the OodleLZ_CompressOptions.
|
Return Value
| return | handle to the async job, or 0 if it was run synchronously
|
Discussion
RunJob will call fp_job(job_data)
it may be done on a thread, or it may run the function synchronously and return 0, indicating the job is already done.
The returned OO_U64 is a handle passed to WaitJob, unless it is 0, in which case WaitJob won't get called.
fp_job should not run until all the dependencies are done. This function should not delete the dependencies.
RunJob must be callable from within an Oodle Job, i.e. jobs may spawn their own sub-jobs directly.
However, the matching WaitJob calls will only ever occur on the thread that called the
internally threaded Oodle API function.
See About Oodle Job Threading Plugins
| OodleXLZ_Decompress_Wide_Async |
|
|
OodleXHandle OodleXLZ_Decompress_Wide_Async( OO_U32 asyncSelect,
const OodleLZ_SeekTable * seekTable,
const void * packedDataPtr,
OO_SINTa packedLen,
void * rawArray,
OO_SINTa rawArrayLen,
OodleLZ_FuzzSafe fuzzSafe OODEFAULT( OodleLZ_FuzzSafe_No ),
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OodleLZ_PackedRawOverlap packedRawOverlap OODEFAULT( OodleLZ_PackedRawOverlap_No ),
OodleXIOQFile writeToFile OODEFAULT( 0 ),
OO_S64 writeToFileStartPos OODEFAULT( 0 ),
OodleXHandle * pWriteHandleGroup OODEFAULT( 0 ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start an async LZ decompress, possibly write raw dataParameters
| asyncSelect | logical OR of OodleXAsyncSelect flags determine how the async is run
|
| seekTable | seek locations as created by OodleLZ_CreateSeekTable
|
| packedDataPtr | pointer to LZ compressed data
|
| packedLen | compressed data length
|
| rawArray | pointer to memory filled with decompressed data
|
| rawArrayLen | length of decompressed data
|
| checkCRC | if OodleLZ_CheckCRC_Yes, the decompressor checks the crc to ensure data integrity
|
| verbosity | (optional) if not OodleLZ_Verbosity_None, will log some information
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder.
|
| decBufSize | (optional) size of circular buffer starting at decBufBase
|
| packedRawOverlap | (optional) if OodleLZ_PackedRawOverlap_Yes, the compressed data is in the same memory array as the output raw data
|
| writeToFile | (optional) OodleXIOQFile to write raw data to
|
| writeToFileStartPos | (optional) file position where writeToFile should start (must be OODLEX_IO_MAX_ALIGNMENT aligned)
|
| pWriteHandleGroup | (optional) if writeToFile is given, this is filled with an OodleAsyncGroup OodleXHandle containing all the file IO operations
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | OodleXHandle to the operation, or OodleXHandle_Null for invalid arguments
|
Discussion
Same as OodleXLZ_ReadAndDecompress_Wide_Async, except this API doesn't include the option to read
the packed data, it must be already fully loaded.
| OodleNetwork1TCP_State_Reset |
|
|
Discussion
Initialize a OodleNetwork1TCP_StateParameters
Discussion
Resets state to a null state.
Generally it is better to make a trained initial state with OodleNetwork1TCP_Train
and then use OodleNetwork1TCP_State_InitAsCopy.
| Getting Started with Oodle Network |
|
|
Oodle Network compresses packets for bandwidth reduction and improved player experience in networked
games. It works with both TCP and UDP.
To compress packages or downloadable content, see Getting Started with Oodle LZ Data Compression
If you are using the Unreal Engine, ask us about our Unreal integration.
Oodle Network does most of its work in an offline training phase. You capture a large sample of
packets. It uses that capture to build a model of your data and learns how to compress it. That
model is saved to disk, and will be shipped with your game.
The runtime component loads the saved model and uses it to compress packets on the fly.
The trained model is what makes it work even on UDP packets, and even on very small packets.
The compression of packets is zero-latency (no extra buffering is introduced). UDP mode compresses
packets independently. TCP mode learns from the previous data sent over each connection.
example_network_client : Example with simple network client support provides a simple example of how you would use a previously trained model to
compress packets in your shipping game.
To evalute Oodle for network packet compression, you need to first get a capture of your typical
network stream to test on.
You should try to capture packets from a realistic game run (perhaps from your QA department
playing the game). The captured packets should be without any encryption or other compression
algorithms applied. You may continue to use your bit-packing or delta scheme. Try to
capture 100 MB of packet data.
The more realistic and varied your packet capture is, the more the test will reflect real world
performance. The capture should not include network protocol headers.
The packet capture format that example_packet loads is :
packet.bin :
U32 [LE] : numbers of channels (num_channels)
repeatedly :
{
U32 [LE] : channel index in [0,num_channels-1]
U32 [LE] : number of bytes of data in this packet (num_bytes)
U8 * num_bytes : payload of this packet
}
// (for UDP set num_channels to 1 and all the channel indices to 0)
Once you have a capture, you can run example_packet : Example demonstrating network packet compression to test Oodle's compressors on it.
Or you can send to us, and we'll run the tests for you. Contact oodle@radgametools.com for details.
NOTE : captured packets should have any existing compression and encryption disabled. Packets should not
contain IP headers. Usually it is best to log the packet data from your network code, don't try to use
an external packet logger. If possible, log packets before splitting them into MTU units. The goal is to
log the raw packet payloads.
See Capturing Training data for OodleNetwork.
What example_packet does is load the captured packets, and splits them into a training set and a testing
set. It then trains a model (TestOodleNetwork_SelectDictionaryAndTrain), and saves it to disk
(OodleNetwork1_Compressor_WriteToFile). It then loads the model from disk, as the game runtime would
(OodleNetwork1_Compressor_LoadFromFileData), and then tests the performance of the model on the
test holdout packets (TestOodleNetwork1UDPPacketCoder_Transmission).
The APIs used for the actual runtime packet compression are OodleNetwork1UDP_Encode and
OodleNetwork1UDP_Decode . (for UDP, similar variants for TCP; toggle testing UDP or TCP with
a #define in example_packet).
See example_packet : Example demonstrating network packet compression for details. It provides a model trainer which you can either use directly, or use as a
starting point for incorporating your own variant in your own tools.
See example_network_client : Example with simple network client support for example of how to use Oodle Network in your shipping runtime.
For more, see About OodleNetwork1, also Capturing Training data for OodleNetwork and Forming Packets for Maximum Compression
For a guide to how to build with the Oodle library on your platform, see About Oodle on Platforms.
For information about the Core vs Ext libs see Oodle2 Core vs Oodle2 Ext.
| OodleNetwork1UDP_State_Size |
|
|
Discussion
Returns the size of memory required for an OodleNetwork1UDP_State objectDiscussion
Shared and State are allocated with malloc( Size() )
| OodleDecompressCallbackRet |
|
|
enum OodleDecompressCallbackRet
{
OodleDecompressCallbackRet_Continue = 0,
OodleDecompressCallbackRet_Cancel = 1,
OodleDecompressCallbackRet_Invalid = 2,
OodleDecompressCallbackRet_Force32 = 0x40000000
};
Discussion
Return value for OodleDecompressCallback
return OodleDecompressCallbackRet_Cancel to abort the in-progress decompression
Discussion
Minimum decompression quantum (for old legacy codecs only)Discussion
Deprecated.
The new sea monster family of compressors use a whole block quantum (OODLELZ_BLOCK_LEN).
Check OodleLZ_Compressor_UsesWholeBlockQuantum
| OodleX_Shutdown_NoThreads |
|
|
Discussion
Shut down Oodle at app exit time.Parameters
| threadProfileLogName | (optional) if not NULL, and the ThreadProfiler is enabled, writes the threadprofiler output to this file name
|
| logLeaks | (optional) if true and the LeakTracker is enabled, logs any leaks or memory or handles
|
| allocStartCounter | (optional) initial counter for the LeakTrack log
|
| debugBreakOnLeaks | (optional) if there are any leaks, do a debug break
|
Discussion
Pair with OodleX_Init_NoThreads. No Oodle functions should be called after Shutdown.
Call Shutdown from the same thread that called Init.
Do not shutdown Oodle then init again. Only call Init and Shutdown once per run.
| OodleXMalloc_GetVTable_Clib |
|
|
Discussion
get an OodleMalloc VTable that contains allocators based on the std clib malloc/free
Discussion
Make a relative path absolute by prefixing the current dirParameters
| addTo | string which will be modified
|
| addToSize | total size of addTo (not the strlen)
|
Discussion
If addTo is an absolute path already it is not changed. If it is relative, the
OS cwd is retrieved with OodleX_GetOSCwd and then prefixed in front of addTo.
| Oodle2 Network API Documentation |
|
|
Discussion
Start a renamefile request.Parameters
| fm | the file to rename (VFS, UTF-8)
|
| to | the new file name (VFS, UTF-8)
|
| overwrite | if true, any existing file of name "to" will be overwrriten
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| priority | (optional) priority of the operation ; see OodleXPriority
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Asynchronously rename a file.
Use OodleXIOQ_ForceWriteable_Async (on the to name) before the rename to force the ovewriting of read-only and other no-access conditions.
Oodle for PS4 is provided as a lib.
lib/liboo2coreps4.a
lib/liboo2extps4.a
The debug build of the Oodle lib is also provided. Generally the release build of Oodle should be linked
with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems.
OodleX file IO on PS4 goes through the sceKernel low level routines, not FIOS.
OodleX Malloc on PS4 uses sceKernelAllocateDirectMemory for large allocations, and clib malloc for small ones.
NOTE : to build the examples on PS4 without worrying about running out of memory, I use :
ifdef _RADPS4_
unsigned int sceLibcHeapExtendedAlloc = 1; /* Switch to dynamic allocation */
size_t sceLibcHeapSize = SCE_LIBC_HEAP_SIZE_EXTENDED_ALLOC_NO_LIMIT; /* no upper limit for heap area */
endif
Do not decompress directly into uncached graphics memory.
See FAQ: How do I decompress to graphics memory quickly?.
Oodle now automatically detects this and warns about it.
Threading utilities.
| OodleCore_Plugins_SetPrintf |
|
|
Discussion
Install the callback used by Oodle Core for loggingParameters
| fp_rrRawPrintf | function pointer to your log function; may be NULL to disable all logging
|
Return Value
| return | returns the previous function pointer
|
Discussion
Use this function to install your own printf for Oodle Core.
The default implementation in debug builds, if you install nothing, uses the C stdio printf for logging.
On Microsoft platforms, it uses OutputDebugString and not stdio.
To disable all logging, call OodleCore_Plugins_SetPrintf(NULL)
WARNING : this function is NOT thread safe! It should be done only once and done in a place where the caller can guarantee thread safety.
In the debug build of Oodle, you can install OodleCore_Plugin_Printf_Verbose to get more verbose logging
| OodleXIOQ_Rename_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_Rename_Async
Discussion
OodleXMalloc_IOAligned result is guaranteed to be aligned to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
| return | pointer to the allocated memory
|
Discussion
OodleXMalloc_IOAligned should be used to get memory that can be used in OodleIOQ and other places
that require disk-aligned pointers.
OodleXMalloc_IOAligned may just pass through to OodleXMallocBig provided by the client, or it may not
if the OodleXMallocBigAlignment is very large.
| OodleX_GetAvailableAsyncSelect |
|
|
Discussion
Get the currently available async systemsReturn Value
Discussion
The OodleXAsyncSelect_Wide bit is set if there is more than one runner available.
| OodleXIOQ_GetLastPendingOpOnFile |
|
|
Discussion
Get an operation on this file, if anyParameters
| file | the IOQFile to query
|
Return Value
| return | the operation found, or 0 if none
|
Discussion
The operation returned may no longer be pending (nor the last) by the time you check it.
| OodleX_S64_to_SINTa_check |
|
|
Discussion
Convert OO_S64 to OO_SINTa and checkDiscussion
Used for loading 64-bit file sizes into memory buffers.
Converts type and checks that file size fits in memory.
In 64 bit builds, this is a no-op.
On 32 bit builds it ensures you to don't lose bits in the cast accidentally.
See also FAQ: What is SINTa? How do I load files bigger than 2 GB?.
| t_fp_OodleNet_Plugin_RunJob |
|
|
Discussion
Function pointer type for OodleNet_Plugins_SetJobSystemParameters
| dependencies | array of handles of other pending jobs. All guaranteed to be nonzero.
|
| num_dependencies | number of dependencies. Guaranteed to be no more than OODLE_JOB_MAX_DEPENDENCIES.
|
| user_ptr | is passed through from the OodleLZ_CompressOptions.
|
Return Value
| return | handle to the async job, or 0 if it was run synchronously
|
Discussion
RunJob will call fp_job(job_data)
it may be done on a thread, or it may run the function synchronously and return 0, indicating the job is already done.
The returned OO_U64 is a handle passed to WaitJob, unless it is 0, in which case WaitJob won't get called.
fp_job should not run until all the dependencies are done. This function should not delete the dependencies.
RunJob must be callable from within an Oodle Job, i.e. jobs may spawn their own sub-jobs directly.
However, the matching WaitJob calls will only ever occur on the thread that called the
internally threaded Oodle API function.
See About Oodle Job Threading Plugins
| FAQ: How do I decompress to graphics memory quickly? |
|
|
The OodleLZ decompressors must read back from the buffer that they are decompressing into.
If you try to decompress directly into uncached memory (or write combined memory, such as graphics buffers),
you may see very bad performance. Several of Oodle's supported targets don't offer an efficient way to
check memory types underlying virtual addresses; instead, Oodle performs a quick check to determine how
fast repeated byte reads from the same location (the first byte of the destination buffer) are and logs
a warning when they are repeatedly very slow. Generally, this means the destination is uncached.
There are a few ways to decode for graphics, depending on your platform and exact usage.
The simplest way is to decompress into a temporary buffer of normal cached memory, and then memcpy from there into the
graphics memory.
For unbounded-window LZ, you would decode into a cached buffer equal to the size of the target. You can copy this buffer
to the final graphics memory either all at once after decoding, or incrementally after each LZ BLOCK or QUANTUM decode from DecodeSome.
You can also use a bounded window by setting the dictionarySize
CompressOption, in which case you can use a sliding window of cached CPU memory, and copy out chunks to the final
target graphics memory as you decode.
See lz_test_11 in example_lz : Example demonstrating LZ compression and decompression for a demonstration of this style of decompression.
On platforms where it is possible, it can be fastest to set your buffer memory type to cached,
do the Oodle decompression into that buffer, then flush the cache, then change the memory type to uncached/GPU.
You must ensure the buffer is not sent to the GPU until the cache flush is done.
When loading levels, ideally you'd do lots of LZ decodes and then only flush the cache once, not after every
resource is decoded.
On the PS4 you could decode into WB_ONION memory, and after the decode is complete, change the memory to WB_GARLIC
using sceKernelMtypeprotect or sceKernelBatchMap.
On the Xbox One you can reserve a range of memory with VirtualAlloc, commit it as READWRITE and cached on the CPU
to do the LZ decode, then decommit the memory and commit the same address again as GPU_READONLY. This is a workaround
which allows you to change the cache type of a memory range (because VirtualProtect does not allow it).
Contact Oodle support for help if you wish to explore these advanced memory type manipulations.
NOTE : on some platforms, Oodle now automatically detects it if you try to decode into uncached memory,
and logs a usage warning about it. See Oodle_SetUsageWarnings
enum OodleLZSeekTable_Flags
{
OodleLZSeekTable_Flags_None = 0,
OodleLZSeekTable_Flags_MakeRawCRCs = 1,
OodleLZSeekTable_Flags_Force32 = 0x40000000
};
Discussion
Options for OodleLZ_CreateSeekTableEnumerants
| OODLENETWORK1_HASH_BITS_DEFAULT |
|
|
Discussion
Good default value for OodleNetwork1 hash table size log2
Discussion
OodleXHandle to a special always-done handle.
Calls to OodleX_GetStatus on this handle value will return &OodleXStatus_Done.
This handle must not be deleted! Do not call OodleX_Wait on it with deleteIfDone = true.
Discussion
Start a make dir request.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Asynchronously make a dir.
| OodleXIOQ_FreeBufferIOAligned_Async |
|
|
Discussion
Start a free-buffer requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Runs OodleXFree_IOAligned as an IO operation on the file's sequential list of ops.
This is mainly useful with OodleXIOQ_Write_Async. When you write a buffer, you can't free it until the write
is done, with this you can just call Write and then FreeBuffer ; the FreeBuffer will be run when the Write is done.
| OodleXLog_SetVerboseLevel |
|
|
Discussion
Set the global verbose levelParameters
Return Value
| return | the previous verbose level
|
Discussion
The verbose level should typically be one of OodleXLog_VerboseLevel ,
but that's not required.
If the verbose level is N, calls to OodleXLog_Printf(M,..) will be
skipped any time M > N. eg. at the default development level of
OodleXLog_Verbose_Some (1), all logs will be output except for
OodleXLog_Printf_v2.
| OodleX low level async io |
|
|
OodleIOQ low level, stateless, asychronous IO.
OodleNetwork1 is a specialized compressor designed for compression of network packet transmission.
It's designed to reduce game bandwidth use and improve player experience.
Quick Intro to OodleNetwork features
- OodleNetwork encodes packet by packet with zero latency. The runtime encode and decode are simple buffer->buffer
calls. Just build your packet as usual and then compress it, and send the compressed packet.
- OodleNetwork runtime integration is extremely simple, just call Encode() in the server and Decode() in the client
(for one-way compression). The complexity is all in the offline training process.
- OodleNetwork does its heavy work in the offline training phase, so the runtime can be quite fast.
- OodleNetwork does no allocations at runtime (you provide the memory for the model). OodleNetwork for UDP uses zero memory per channel. OodleNetwork for TCP uses around 100k per channel, which may be provided by the client code as a linear block.
- OodleNetwork is about the same speed as "zlib -5" or "LZ4 HC" while achieving much more compression.
- OodleNetwork is able to compress packets even if they have already been bit-packed.
- OodleNetwork will never expand a packet.
- OodleNetwork uses a dictionary size of your choosing. Even a small 1 MB dictionary can provide good compression.
A typical size is 4 MB. Larger dictionaries generally give more compression, but you are free to choose the tradeoff
that best suits your needs.
- OodleNetwork is in Oodle2 Core and the runtime available on all platforms. (model training can only be done on desktop platforms)
- OodleNetwork is fully thread-safe and re-entrant with no mutexes or blocking. The shared model data is read-only
after initialization.
Compression of UDP network packets
With UDP networking, packets may be dropped or arrive in different orders. This means there is
not a consistent history that the encoder and decoder see, so you cannot use any compressor which
is based on a per-channel history (zlib, OodleLZ, OodleNetwork1-TCP).
OodleNetwork1UDP does compression of packets without any per-channel state.
The OodleNetwork1UDP_State is global and const; you make it offline in training, then distribute it as a const block of data with your game. It can be used by all encoder and decoder channels.
Sometimes it may be advantageous to have separate models for upstream or downstream traffic, or to compress only one direction and not the other. You can easily try these options and choose what works best for you.
Compression is done on each packet independently, so they can be lost or out of order and still decompress correctly.
OodleNetwork1 [dictionary MB|hash bits]
OodleNetwork1 for TCP with per-channel history :
OodleNetwork1 [8|19] : 595654217 -> 123101634 = 4.839:1
1605378 packets; 371.0 -> 76.7 average
OodleNetwork1 for UDP with no per-channel state :
OodleNetwork1UDP [8|19] : 595654217 -> 150022411 = 3.970:1
1605378 packets; 371.0 -> 93.4 average
Compression of TCP network packets
NOTE : OodleNetwork1 and OodleLZ are for TCP network compression, that is when
you have a reliable per-channel history. For UDP, use OodleNetwork1UDP (see below).
OodleNetwork1 for TCP uses a shared static dictionary, plus a dynamic state per channel.
Usage of OodleNetwork1 is demonstrated in example_packet : Example demonstrating network packet compression.
For more normal data distribution needs, and large downloads or join packets, use Oodle Data Compression. See About OodleLZ.
OodleNetwork1 is "zero latency". That is, it don't add any buffering or delay of packets. It produces compressed bytes immediately for each raw byte processed.
OodleNetwork1 uses a static dictionary and hash table which is const and shared by all network channels. The size is set by the user. The bigger the static dictionary, the more compression you will get. There is an adaptive per-channel arithmetic coder so that the match length and literal statistics can adapt to the channel a bit (this was a big win vs. using any kind of static models).
OodleNetwork1 has only 104k of per-channel state. (compare to zlib which uses 400k per encoder)
On the server, a large static dictionary is no problem. They're running 16GB servers with 10,000 connections, they really don't care if the static dictionary is 64MB. However, that same static dictionary also has to be on the client, so the limit on how big a static dictionary you can use really comes from the client side. I suspect that something in the 8MB - 16MB range is reasonable. (and of course you can compress the static dictionary; it's only something like 2-4 MB that you have to distribute and load). (for loading the compressed static dictionary off disk, see About OodleLZ)
(BTW you don't necessarily need an adaptive compression state for every open channel. If some channels tend to go idle, you could drop their state. When the channel starts up again, grab a fresh state (and send a reset message to the client so it wipes its adaptive state). You could do something like have a few thousand compression states which you cycle in an LRU for an unbounded number of open channels. Of course the problem with that is if you actually get a higher number of simultaneous active connections you would be recycling states all the time, which is just the standard cache over-commit problem that causes nasty thrashing, so YMMV etc.)
Here are some real world results :
OodleNetwork1 TCP [dictionary MB|hash bits]
OodleNetwork1 [4|18] : 595654217 -> 131935361 = 4.515:1
1605378 packets; 371.0 -> 82.2 average
OodleNetwork1 [8|19] : 595654217 -> 123101634 = 4.839:1
1605378 packets; 371.0 -> 76.7 average
OodleNetwork1 [16|20] : 595654217 -> 110427772 = 5.394:1
1605378 packets; 371.0 -> 68.8 average
OodleNetwork1 [32|21] : 595654217 -> 93276137 = 6.386:1
1605378 packets; 371.0 -> 58.1 average
See OodleAPI_OodleNetwork1 , also Capturing Training data for OodleNetwork and Forming Packets for Maximum Compression
| OodleX_Init_GetDefaults_DebugSystems |
|
|
enum OodleX_Init_GetDefaults_DebugSystems
{
OodleX_Init_GetDefaults_DebugSystems_No = 0,
OodleX_Init_GetDefaults_DebugSystems_Yes = 1,
OodleX_Init_GetDefaults_DebugSystems_Force32 = 0x40000000
};
Discussion
Should GetDefaults enable debugging systems?
enum OodleLZ_Compressor
{
OodleLZ_Compressor_Invalid = -1,
OodleLZ_Compressor_None = 3,
OodleLZ_Compressor_Kraken = 8,
OodleLZ_Compressor_Leviathan = 13,
OodleLZ_Compressor_Mermaid = 9,
OodleLZ_Compressor_Selkie = 11,
OodleLZ_Compressor_Hydra = 12,
OodleLZ_Compressor_BitKnit = 10,
OodleLZ_Compressor_LZB16 = 4,
OodleLZ_Compressor_LZNA = 7,
OodleLZ_Compressor_LZH = 0,
OodleLZ_Compressor_LZHLW = 1,
OodleLZ_Compressor_LZNIB = 2,
OodleLZ_Compressor_LZBLW = 5,
OodleLZ_Compressor_LZA = 6,
OodleLZ_Compressor_Count = 14,
OodleLZ_Compressor_Force32 = 0x40000000
};
Discussion
Selection of compression algorithm.Enumerants
Discussion
Each compressor provides a different balance of speed vs compression ratio.
New Oodle users should only use the new sea monster family of compressors.
The OODLE_ALLOW_DEPRECATED_COMPRESSORS set of compressors is no longer supported
as of Oodle 2.9.0 ; see FAQ: What are the Oodle deprecated compressors ?
The sea monsters are all fuzz safe and use whole-block quantum (not the 16k quantum)
(OodleLZ_Compressor_UsesWholeBlockQuantum)
If you need to encode the deprecated compressors, define OODLE_ALLOW_DEPRECATED_COMPRESSORS before
including oodle2.h
See FAQ: Which OodleLZ should I use? for a quick FAQ on which compressor to use
See About OodleLZ for discussion of how to choose a compressor.
struct OodleLZ_CompressOptions
{
OO_U32 unused_was_verbosity;
OO_S32 minMatchLen;
OO_BOOL seekChunkReset;
OO_S32 seekChunkLen;
OodleLZ_Profile profile;
OO_S32 dictionarySize;
OO_S32 spaceSpeedTradeoffBytes;
OO_S32 unused_was_maxHuffmansPerChunk;
OO_BOOL sendQuantumCRCs;
OO_S32 maxLocalDictionarySize;
OO_BOOL makeLongRangeMatcher;
OO_S32 matchTableSizeLog2;
OodleLZ_Jobify jobify;
void * jobifyUserPtr;
OO_S32 farMatchMinLen;
OO_S32 farMatchOffsetLog2;
OO_U32 reserved[4];
};
Discussion
Options for the compressorMembers
| unused_was_verbosity | unused ; was verbosity (set to zero) |
| minMatchLen | minimum match length ; cannot be used to reduce a compressor's default MML, but can be higher. On some types of data, a large MML (6 or 8) is a space-speed win. |
| seekChunkReset | whether chunks should be independent, for seeking and parallelism |
| seekChunkLen | length of independent seek chunks (if seekChunkReset) ; must be a power of 2 and >= OODLELZ_BLOCK_LEN ; you can use OodleLZ_MakeSeekChunkLen |
| profile | decoder profile to target (set to zero) |
| dictionarySize | sets a maximum offset for matches, if lower than the maximum the format supports. <= 0 means infinite (use whole buffer). Often power of 2 but doesn't have to be. |
| spaceSpeedTradeoffBytes | this is a number of bytes; I must gain at least this many bytes of compressed size to accept a speed-decreasing decision |
| unused_was_maxHuffmansPerChunk | unused ; was maxHuffmansPerChunk |
| sendQuantumCRCs | should the encoder send a CRC of each compressed quantum, for integrity checks; this is necessary if you want to use OodleLZ_CheckCRC_Yes on decode |
| maxLocalDictionarySize | (Optimals) size of local dictionary before needing a long range matcher. This does not set a window size for the decoder; it's useful to limit memory use and time taken in the encoder. maxLocalDictionarySize must be a power of 2. Must be <= OODLELZ_LOCALDICTIONARYSIZE_MAX |
| makeLongRangeMatcher | (Optimals) should the encoder find matches beyond maxLocalDictionarySize using an LRM |
| matchTableSizeLog2 | (non-Optimals) when variable, sets the size of the match finder structure (often a hash table) ; use 0 for the compressor's default |
| jobify | controls internal job usage by compressors |
| jobifyUserPtr | user pointer passed through to RunJob and WaitJob callbacks |
| farMatchMinLen | far matches must be at least this len |
| farMatchOffsetLog2 | if not zero, the log2 of an offset that must meet farMatchMinLen |
| reserved | reserved space for adding more options; zero these! |
Discussion
Typically filled by calling OodleLZ_CompressOptions_GetDefault , then individual options may be modified, like :
OodleLZ_CompressOptions my_options = OodleLZ_CompressOptions_GetDefault()
To ensure you have set up the options correctly, call OodleLZ_CompressOptions_Validate.
unused_was_verbosity : place holder, set to zero
minMatchLen : rarely useful. Default value of 0 means let the compressor decide. On some types of data,
bumping this up to 4,6, or 8 can improve decode speed with little effect on compression ratio. Most of the
Oodle compressors use a default MML of 4 at levels below 7, and MML 3 at levels >= 7. If you want to keep MML 4
at the higher levels, set minMatchLen here to 4. minMatchLen cannot be used to reduce the base MML of the compressor, only to increase it.
seekChunkReset must be true if you want the decode to be able to run "Wide", with pieces that can be
decoded independently (not keeping previous pieces in memory for match references).
seekChunkLen : length of independent seek chunks (if seekChunkReset) ; must be a power of 2 and >= OODLELZ_BLOCK_LEN ; you can use OodleLZ_MakeSeekChunkLen
profile : tells the encoder to target alternate bitstream profile. Default value of zero for normal use.
dictionarySize : limits the encoder to partial buffer access for matches. Can be useful for decoding incrementally
without keeping the entire output buffer in memory.
spaceSpeedTradeoffBytes is a way to trade off compression ratio for decode speed. If you make it smaller,
you get more compression ratio and slower decodes. It's the number of bytes that a decision must save to
be worth a slower decode. Default is 256 (OODLELZ_SPACESPEEDTRADEOFFBYTES_DEFAULT). So that means the encoder must be able to save >= 256 bytes to
accept something that will slow down decoding (like adding another Huffman table). The typical range is
64-1024.
Lower spaceSpeedTradeoffBytes = more compression, slower decode
Higher spaceSpeedTradeoffBytes = less compression, faster decode
spaceSpeedTradeoffBytes is the primary parameter for controlling Hydra. The default value of 256 will make
Hydra decodes that are just a little bit faster than Kraken. You get Kraken speeds around 200, and Mermaid
speeds around 1200.
At the extreme, a spaceSpeedTradeoffBytes of zero would mean all you care about is compression ratio, not decode
speed, you want the encoder to make the smallest possible output. (you cannot actually set zero, as zero values
always mean "use default" in this struct; you never really want zero anyway)
Generally spaceSpeedTradeoffBytes below 16 provides diminishing gains in size with pointless decode speed loss.
spaceSpeedTradeoffBytes is on sort of powers of 2 scale, so you might want to experiment with 32,64,128,256,512
spaceSpeedTradeoffBytes outside the range [16 - 2048] is not recommended.
unused_was_maxHuffmansPerChunk : place holder, set to zero
sendQuantumCRCs : send hashes of the compressed data to verify in the decoder; not recommended, if you need data
verification, use your own system outside of Oodle. DEPRECATED, not recommended. For backwards compatibility only.
maxLocalDictionarySize : only applies to optimal parsers at level >= Optimal2. This limits the encoder memory use.
Making it larger = more compression, higher memory use. Matches within maxLocalDictionarySize are found exactly,
outside the maxLocalDictionarySize window an approximate long range matcher is used.
makeLongRangeMatcher : whether an LRM should be used to find matches outside the maxLocalDictionarySize window
(Optimal levels only)
matchTableSizeLog2 : for non-optimal levels (level <= Normal), controls the hash table size. Making this very
small can sometimes boost encoder speed. For the very fastest encoding, use the SuperFast level and change
matchTableSizeLog2 to 12 or 13.
matchTableSizeLog2 should usually be left zero to use the encoder's default
matchTableSizeLog2 allows you to limit memory use of the non-Optimal encoder levels. Memory use is roughly
( 1 MB + 4 << matchTableSizeLog2 )
jobify tells compressors how to use internal jobs for compression tasks. Jobs can be run in parallel using the
job system plugins set with OodleCore_Plugins_SetJobSystem. Not all compressors or compression level support
jobs, but the slower ones generally do. The default value of jobify is to use a thread system if one is installed.
farMatchMinLen and farMatchOffsetLog2 can be used to tune the encoded stream for a known cache size on the
decoding hardware. If set, then offsets with log2 greater or each to farMatchOffsetLog2 must have a minimum
length of farMatchMinLen. For example to target a machine with a 2 MB cache, set farMatchOffsetLog2 to 21,
and farMatchMinLen to something large, like 16 or 20.
Without farMatchMinLen and farMatchOffsetLog2 set, the Oodle encoders tune for a blend of cache sizes that works
well on most machines. dictionarySize can also be used to tune for cache size, but cuts off all matches
beyond a certain distance. That may be more appropriate when you don't want to go out of cache at all.
farMatchMinLen can only be used to make the standard blend target more restrictive; it can reduce the target cache size
but can't make it larger (or it can raise min match len outside cache but can't make it shorter).
For help on setting up OodleLZ_CompressOptions contact support at oodle@radgametools.com
NOTE : fields you do not set should always be zero initialized. In particular the reserved fields should be zeroed.
Zero always means "use default" and is a future-portable initialization value.
If you set fields to zero to mean "use default" you can call OodleLZ_CompressOptions_Validate to change them
to default values. This is done automatically internally if you don't do it explicitly.
| OodleXIOQ_Delete_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_Delete_Async
Discussion
Get the Status of a request, and optionally delete if doneParameters
| req | the IOQ operation handle to work on
|
| andDeleteIfDone | if true and the returned status is >= Done the handle will be deleted
|
| pErrorCode | (optional) the OS error code, if any
|
| pReturnValue | (optional) the operation return value
|
Return Value
| return | the status of the request
|
Discussion
This function is similar to OodleX_GetStatus, but for IOQ operation handles only, and it provides
more information (optionally).
The error code returned can be processed with OodleXIOQ_GetErrorEnum or OodleXIOQ_GetErrorDetails.
The return value depends on the operation type. For example if the operation is a Read, it returns the
number of bytes successfully read.
| OODLEX_ASYNC_HANDLE_INVALID |
|
|
Discussion
OodleXHandle for an invalid handle.
Calls to OodleX_GetStatus on this handle value will return &OodleXStatus_Invalid.
Oodle on IOS is currently provided as only the Oodle Core lib (no OodleX).
This includes the synchronous LZ compressors, as well as Oodle Network compression.
See Oodle2 Core vs Oodle2 Ext
Oodle on mobile does not include the Optimal level encoders. When OodleLZ_CompressionLevel_Optimal1 or higher
is request, OodleLZ_CompressionLevel_Normal is used instead.
Discussion
Get the OodleXLogCallbackRet currently setReturn Value
| return | the global callback currently set
|
Discussion
See OodleXLog_SetCallback
| example_packet : Example demonstrating network packet compression |
|
|
Discussion
Oodle example_packet
Example demonstrating network packet compression.
(See also example_network_client : Example with simple network client support for a simpler client-only example that's more
suitable as a basis for your runtime)
The primary API is OodleNetwork1 (see About OodleNetwork1 and OodleAPI_OodleNetwork1).
For an overview see About Oodle Network Compression
To use example_packet, you must capture a sample of your game's network packets.
See Capturing Training data for OodleNetwork
This example can be used with Oodle Network lib only. If you wish to also apply LZ
compression to the save model runtime data, toggle USE_OODLE_LZ_DATA_COMPRESSION.
#include "../include/oodle2net.h"
// optional, use Oodle LZ for the network model data
//#define USE_OODLE_LZ_DATA_COMPRESSION
#if defined(USE_OODLE_LZ_DATA_COMPRESSION) && ! defined(__OODLE2_H_INCLUDED__)
#include "../include/oodle2x.h"
#endif
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#if 1
// if the STL build gives you trouble, then set this to 0
// this is just used for extra diagnostic printing
#define EXAMPLE_PACKET_DO_SORT 1
#else
#define EXAMPLE_PACKET_DO_SORT 0
#endif
#define _ITERATOR_DEBUG_LEVEL 0
#define _HAS_ITERATOR_DEBUGGING 0
#if EXAMPLE_PACKET_DO_SORT
// try to get std::sort
// this is used for information display only
#include <algorithm>
#endif
#include "ooex.h" // example helpers
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_packet
#endif
#ifdef _MSC_VER
#pragma warning(disable: 4505)
#endif
#ifndef __OODLE2_H_INCLUDED__
// if USE_OODLE_LZ_DATA_COMPRESSION off,
// make a fake define of OodleLZ_Compressor_Invalid
// so I can share the code path but toggle compression off
typedef enum OodleLZ_Compressor
{
OodleLZ_Compressor_Invalid = -1,
OodleLZ_Compressor_Force32 = 0x40000000
} OodleLZ_Compressor;
#endif
//===========================================================
This is where you #define options to select which compressors the example should test
To test OodleNetwork1
use EXAMPLE_PACKET_TEST_OODLENETWORK1_TCP for TCP networking
or EXAMPLE_PACKET_TEST_OODLENETWORK1_UDP for UDP networking
If you set EXAMPLE_PACKET_TEST_OODLENETWORK1_DIC_SIZES, a variety of OodleNetwork1 dictionary sizes are tested.
// test OodleNetwork for UDP or TCP :
#define EXAMPLE_PACKET_TEST_OODLENETWORK1_TCP 0
#define EXAMPLE_PACKET_TEST_OODLENETWORK1_UDP 1
// test multiple dictionary sizes and report results
#define EXAMPLE_PACKET_TEST_OODLENETWORK1_DIC_SIZES 0
// enable Load Existing to repeat a test without re-training
#define EXAMPLE_PACKET_OODLENETWORK_LOAD_EXISTING 0
// Trials is slow but gives more compression
#define EXAMPLE_PACKET_OODLENETWORK_DO_TRIALS 0
//===========================================================
// get a vector :
#include <stddef.h>
#ifdef __GNUC__
#include <stdint.h>
#endif
#include <new> // for operator new placement new
#define CB_ALLOC(size) malloc(size)
#define CB_FREE(ptr,size) free(ptr)
#define CB_ASSERT_MALLOC(exp) OOEX_ASSERT_ALWAYS(exp)
#include "cbvector.h"
//===========================================================
// little helper object to load a file and free it for us
#include "read_whole_file.h"
static OO_BOOL write_whole_file(const char * file_name,void * ptr,OO_SINTa size)
{
FILE * fp = fopen(file_name,"wb");
if ( ! fp ) return false;
size_t count = fwrite(ptr,1,size,fp);
fclose(fp);
if ( count != (size_t)size ) return false;
return true;
}
#ifndef BUILDING_TEST_PACKET
#define EXAMPLE_PACKET_DEFAULT_OUTPUT_PATH
namespace
{
struct OodleIOBuffer
{
OO_U8 * ptr;
OO_SINTa size;
explicit OodleIOBuffer(const char * name)
{
OodleXLog_Printf_v1("loading : %s ...\n",name);
ptr = (OO_U8 *) read_whole_file(name,&size);
if ( ! ptr )
{
OodleXLog_Printf_v0("failed to read %s\n",name);
size = 0;
}
}
explicit OodleIOBuffer(OO_SINTa alloc_size)
{
ptr = (OO_U8 *) malloc(alloc_size);
size = alloc_size;
}
void Release()
{
if ( ptr )
{
free(ptr);
ptr = NULL;
size = 0;
}
}
~OodleIOBuffer()
{
Release();
}
};
};
#endif
//===========================================================
// protos :
static void TestOodleNetwork1TCPPacketCoder(
const OodleIOBuffer & iob_packet_train ,
const OodleIOBuffer & iob_packet_test ,
int on1_dic_mb , int on1_ht_bits);
static void TestOodleNetwork1UDPPacketCoder(
const OodleIOBuffer & iob_packet_train ,
const OodleIOBuffer & iob_packet_test ,
int on1_dic_mb , int on1_ht_bits,
int for_oodle_version_major);
//===========================================================
File names to find captured packets.
You should change these to the files you have captured.
You may also pass the file name and holdout fraction on the command line.
The holdout fraction should be adjusted to ensure you have enough packets for testing and training.
static const char * c_example_packet_file = "r:\\packet.bin"; // <- change this to your file name
static const int c_example_packet_test_holdout_fraction_denominator = 4; // hold out 1/4
// useful on very large samples :
//static const int c_example_packet_test_holdout_fraction_denominator = 2; // hold out 1/2
Some variables that control filtering of the packet stream, if you wish to reject some :
// to select a subset :
static const int c_packet_filter_skip_initial = 0;
static const int c_packet_filter_max_num = 1<<29;
//static const int c_packet_filter_skip_initial = 2500000;
//static const int c_packet_filter_max_num = 100000;
// to select a range of sizes :
//static const int c_packet_filter_min_size = 0;
//static const int c_packet_filter_max_size = 1<<29;
static const int c_packet_filter_min_size = 32; // exclude tiny packets
static const int c_packet_filter_max_size = 8192; // exclude large packets
example_packet_file should contain a capture of packets from a real game session.
It should be at least 100 MB.
See Capturing Training data for OodleNetwork
Some portion (as set by example_packet_test_holdout_fraction_denominator) will be held out for testing
the compression level. The remainder will be used for training. In real game usage, you don't need to
hold out any for testing, this is just for evaluating the compression level. To be fair, the packets
used in training are not used in testing.
The file format is :
packet.bin :
OO_U32 [LE] : numbers of channels (num_channels)
repeatedly :
{
OO_U32 [LE] : channel index in [0,num_channels-1]
OO_U32 [LE] : number of bytes of data in this packet (num_bytes)
OO_U8 * num_bytes : payload of this packet
}
Channels are for TCP networking. For UDP, set num_channels = 1 and all channel indices to 0.
//===========================================================
static OO_U32 irandmod(OO_U32 size);
static void scan_packet_iobs(
const OodleIOBuffer & iob_packet,
OodleIOBuffer * p_iob_packet_test,
OodleIOBuffer * p_iob_packet_train,
int example_packet_test_holdout_fraction_denominator)
{
const OO_U8 * packet_bin_buf = iob_packet.ptr;
const OO_SINTa packet_bin_size = iob_packet.size;
// get the number of channels ; first dword in packet bin :
const OO_U8 * packet_bin_ptr = packet_bin_buf;
const OO_U8 * packet_bin_end = packet_bin_ptr + packet_bin_size;
OodleIOBuffer & iob_packet_test = * p_iob_packet_test;
OO_U8 * packet_test_ptr = iob_packet_test.ptr;
OO_U8 * packet_train_ptr = p_iob_packet_train ? p_iob_packet_train->ptr : NULL;
OO_S32 num_channels = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
if ( num_channels == 0 ) num_channels = 1;
OOEX_ASSERT( num_channels > 0 );
OOEX_PUT32_LE(packet_test_ptr,num_channels); packet_test_ptr += sizeof(OO_U32);
if ( packet_train_ptr )
{
OOEX_PUT32_LE(packet_train_ptr,num_channels); packet_train_ptr += sizeof(OO_U32);
}
OO_SINTa tot_packet_bytes = 0;
OO_SINTa tot_num_packets = 0;
OO_SINTa train_packet_bytes = 0;
OO_SINTa train_num_packets = 0;
OO_SINTa test_packet_bytes = 0;
OO_SINTa test_num_packets = 0;
OO_SINTa num_packets_excluded = 0;
OO_SINTa packet_bytes_excluded = 0;
OO_SINTa num_packets_initial = 0;
OO_SINTa smallest_packet_size = 9999999;
OO_SINTa largest_packet_size = 0;
const OO_U8 * largest_packet_ptr = NULL;
while( packet_bin_ptr < packet_bin_end )
{
// grab the current packet header :
OO_S32 channel = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OO_S32 bytes = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_ASSERT( channel >= 0 && channel < num_channels );
OOEX_ASSERT( bytes > 0 && bytes < (256*1024) );
if ( num_packets_initial < c_packet_filter_skip_initial )
{
num_packets_initial++;
if ( num_packets_initial == c_packet_filter_skip_initial )
{
OodleXLog_Printf_v1("skipped initial %d packets\n",c_packet_filter_skip_initial);
}
// skip it
packet_bin_ptr += bytes;
num_packets_excluded ++;
packet_bytes_excluded += bytes;
continue;
}
// get largest and smallest packet size *before* size filters :
smallest_packet_size = OOEX_MIN(smallest_packet_size,bytes);
if ( bytes > largest_packet_size )
{
largest_packet_size = bytes;
largest_packet_ptr = packet_bin_ptr;
}
if ( bytes < c_packet_filter_min_size ||
bytes > c_packet_filter_max_size )
{
// skip it
packet_bin_ptr += bytes;
num_packets_excluded ++;
packet_bytes_excluded += bytes;
continue;
}
if ( packet_bin_ptr + bytes > packet_bin_end )
{
// partial end packet
break;
}
tot_num_packets++;
tot_packet_bytes += bytes;
if ( packet_train_ptr == NULL ||
irandmod(example_packet_test_holdout_fraction_denominator) == 0 )
{
// test
OOEX_PUT32_LE(packet_test_ptr,channel); packet_test_ptr += sizeof(OO_S32);
OOEX_PUT32_LE(packet_test_ptr,bytes); packet_test_ptr += sizeof(OO_S32);
memcpy(packet_test_ptr,packet_bin_ptr,bytes);
packet_bin_ptr += bytes;
packet_test_ptr += bytes;
test_num_packets ++;
test_packet_bytes += bytes;
}
else
{
//train
OOEX_PUT32_LE(packet_train_ptr,channel); packet_train_ptr += sizeof(OO_S32);
OOEX_PUT32_LE(packet_train_ptr,bytes); packet_train_ptr += sizeof(OO_S32);
memcpy(packet_train_ptr,packet_bin_ptr,bytes);
packet_bin_ptr += bytes;
packet_train_ptr += bytes;
train_num_packets ++;
train_packet_bytes += bytes;
}
if ( tot_num_packets >= c_packet_filter_max_num )
break;
}
iob_packet_test.size = packet_test_ptr - iob_packet_test.ptr;
if ( p_iob_packet_train )
p_iob_packet_train->size = packet_train_ptr - p_iob_packet_train->ptr;
OodleXLog_Printf_v1("Read %d packets, %lld bytes (%.2f average)\n",(int)tot_num_packets,(long long)tot_packet_bytes,(double)tot_packet_bytes/tot_num_packets);
OodleXLog_Printf_v1("Held out %d packets (%lld bytes) for testing, %d (%lld) for training\n",
(int)test_num_packets,(long long)test_packet_bytes,
(int)train_num_packets,(long long)train_packet_bytes);
if ( num_packets_excluded > 0 )
{
OodleXLog_Printf_v1("%d packets excluded by filters (%d bytes)\n",(int)num_packets_excluded,
(int)packet_bytes_excluded);
}
OodleXLog_Printf_v1("smallest_packet_size : %d (%d min filtered)\n",(int)smallest_packet_size,c_packet_filter_min_size);
OodleXLog_Printf_v1("largest_packet_size : %d (%d max filtered)\n",(int)largest_packet_size,c_packet_filter_max_size);
if ( largest_packet_size > c_packet_filter_max_size )
{
OodleXLog_Printf_v1("WARNING: large packets were found and excluded by filter.\n");
OodleXLog_Printf_v1(" large packets SHOULD be compressed. Consider using OodleLZ.\n");
}
#if 0
// log the largest packet to see what it is
if ( largest_packet_size > 0 )
{
OodleXIOQ_WriteWholeFile_AsyncAndWait("r:\\largest_packet.bin",largest_packet_ptr,largest_packet_size,OodleXFileOpenFlags_Buffered);
}
#else
OOEX_UNUSED_VARIABLE(largest_packet_ptr);
#endif
}
static void test_packet_iobs(const OodleIOBuffer & iob_packet_test,
const OodleIOBuffer & iob_packet_train,
int for_oodle_version_major);
static int example_packet_onefile(const char * example_packet_file,int example_packet_test_holdout_fraction_denominator,int for_oodle_version_major)
{
OodleXLog_Printf_v1("Loading : \"%s\" , holding out 1/%d\n",example_packet_file,example_packet_test_holdout_fraction_denominator);
if ( example_packet_test_holdout_fraction_denominator < 2 )
{
OodleXLog_Printf_v0("Error : invalid holdout fraction, must be >=2\n");
return 10;
}
//=================================================
{
// load the files containing packet data :
OodleIOBuffer iob_packet(example_packet_file);
if ( ! iob_packet.ptr )
{
OodleXLog_Printf_v0("Failed to load : %s\n",example_packet_file);
return 10;
}
//---------------------------------------
// do test/train holdout :
OodleIOBuffer iob_packet_test(iob_packet.size);
OodleIOBuffer iob_packet_train(iob_packet.size);
scan_packet_iobs(iob_packet,&iob_packet_test,&iob_packet_train,example_packet_test_holdout_fraction_denominator);
iob_packet.Release();
test_packet_iobs(iob_packet_test,iob_packet_train,for_oodle_version_major);
}
return 0;
}
#ifdef BUILDING_TEST_PACKET
static int example_packet_twofile(const char * packet_train_file,const char * packet_test_file,int for_oodle_version_major)
{
// load the files containing packet data :
OodleIOBuffer iob_packet_test(packet_test_file);
if ( ! iob_packet_test.ptr )
{
OodleXLog_Printf_v0("Failed to load : %s\n",packet_test_file);
return 10;
}
OodleXLog_Printf_v1("Test file (%s) : %d bytes\n",packet_test_file,(int)iob_packet_test.size);
OodleIOBuffer iob_packet_test_parsed(iob_packet_test.size);
// "scan" to apply exclusions and such :
scan_packet_iobs(iob_packet_test,&iob_packet_test_parsed,NULL,0);
iob_packet_test.Release();
OodleIOBuffer iob_packet_train(packet_train_file);
if ( ! iob_packet_train.ptr )
{
OodleXLog_Printf_v0("Failed to load : %s\n",packet_train_file);
return 10;
}
OodleXLog_Printf_v1("Train file (%s) : %d bytes\n",packet_train_file,(int)iob_packet_train.size);
OodleIOBuffer iob_packet_train_parsed(iob_packet_train.size);
scan_packet_iobs(iob_packet_train,&iob_packet_train_parsed,NULL,0);
iob_packet_train.Release();
test_packet_iobs(iob_packet_test_parsed,iob_packet_train_parsed,for_oodle_version_major);
return 0;
}
#endif
static void test_packet_iobs(const OodleIOBuffer & iob_packet_test,
const OodleIOBuffer & iob_packet_train,
int for_oodle_version_major)
{
//---------------------------
Run one or more tests based on the #defined options :
//---------------------------
if ( EXAMPLE_PACKET_TEST_OODLENETWORK1_TCP )
{
if ( EXAMPLE_PACKET_TEST_OODLENETWORK1_DIC_SIZES )
{
// test several typical on1 dictionary sizes
TestOodleNetwork1TCPPacketCoder(iob_packet_train,iob_packet_test,1,17 );
TestOodleNetwork1TCPPacketCoder(iob_packet_train,iob_packet_test,2,18 );
TestOodleNetwork1TCPPacketCoder(iob_packet_train,iob_packet_test,4,19 );
TestOodleNetwork1TCPPacketCoder(iob_packet_train,iob_packet_test,8,20 );
TestOodleNetwork1TCPPacketCoder(iob_packet_train,iob_packet_test,16,21 );
}
else
{
int on1_dictionary_mb = 4;
int on1_ht_bits = 18;
//int on1_dictionary_mb = 8;
//int on1_ht_bits = 19;
//int on1_dictionary_mb = 16;
//int on1_ht_bits = 20;
TestOodleNetwork1TCPPacketCoder(
iob_packet_train,
iob_packet_test,
on1_dictionary_mb,
on1_ht_bits );
}
}
//---------------------------
if ( EXAMPLE_PACKET_TEST_OODLENETWORK1_UDP )
{
if ( EXAMPLE_PACKET_TEST_OODLENETWORK1_DIC_SIZES )
{
//OodleXLog_SetVerboseLevel(0);
TestOodleNetwork1UDPPacketCoder( iob_packet_train,iob_packet_test,1,17 ,for_oodle_version_major);
TestOodleNetwork1UDPPacketCoder( iob_packet_train,iob_packet_test,2,18 ,for_oodle_version_major);
TestOodleNetwork1UDPPacketCoder( iob_packet_train,iob_packet_test,4,19 ,for_oodle_version_major);
TestOodleNetwork1UDPPacketCoder( iob_packet_train,iob_packet_test,8,20 ,for_oodle_version_major);
TestOodleNetwork1UDPPacketCoder( iob_packet_train,iob_packet_test,16,21 ,for_oodle_version_major);
// 16 MB dictionary is max due to 24 bit index
}
else
{
//int on1_dictionary_mb = 1;
//int on1_ht_bits = 17;
int on1_dictionary_mb = 4;
int on1_ht_bits = 19;
//int on1_dictionary_mb = 8;
//int on1_ht_bits = 19;
//int on1_dictionary_mb = 16;
//int on1_ht_bits = 20;
TestOodleNetwork1UDPPacketCoder(
iob_packet_train,
iob_packet_test,
on1_dictionary_mb,
on1_ht_bits ,
for_oodle_version_major);
}
}
//---------------------------
}
#ifdef __OODLE2X_H_INCLUDED__
static void Install_OodleX_Plugins_to_OodleNet()
{
OodleNet_Plugins_SetPrintf(OodleXLog_Printf_Raw);
OodleNet_Plugins_SetAssertion(OodleX_DisplayAssertion);
OodleNet_Plugins_SetAllocators(OodleXMallocAligned,OodleXFree);
OodleNet_Plugins_SetJobSystemAndCount(OodleX_CorePlugin_RunJob,OodleX_CorePlugin_WaitJob,OodleX_GetNumWorkerThreads());
}
#endif
extern "C" int main(int argc,char *argv[])
{
#ifdef __OODLE2X_H_INCLUDED__
// Init Oodle systems with default options :
if ( ! OodleX_Init_Default(OODLE_HEADER_VERSION) )
{
fprintf(stderr,"OodleX_Init failed.\n");
return 10;
}
Install_OodleX_Plugins_to_OodleNet();
#endif
OodleXLog_Printf_v1("example_packet [packet_file] [test_holdout] [for_oodle_version_major]\n");
// defaults :
const char * example_packet_file = c_example_packet_file;
int example_packet_test_holdout_fraction_denominator = c_example_packet_test_holdout_fraction_denominator;
int for_oodle_version_major = OODLE2NET_VERSION_MAJOR;
// overrides in args :
if ( argc >= 2 )
{
example_packet_file = argv[1];
}
if ( argc >= 3 )
{
example_packet_test_holdout_fraction_denominator = atoi(argv[2]);
OodleXLog_Printf_v1("example_packet_test_holdout_fraction_denominator = %d\n",example_packet_test_holdout_fraction_denominator);
}
#ifndef BUILDING_TEST_PACKET
if ( argc >= 4 )
{
for_oodle_version_major = atoi(argv[3]);
OodleXLog_Printf_v1("for_oodle_version_major = %d\n",for_oodle_version_major);
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
// set LZ back-compat version as well :
struct OodleConfigValues oodleconfig;
Oodle_GetConfigValues(&oodleconfig);
oodleconfig.m_OodleLZ_BackwardsCompatible_MajorVersion = for_oodle_version_major;
Oodle_SetConfigValues(&oodleconfig);
#endif
}
#endif
int ret = example_packet_onefile(example_packet_file,example_packet_test_holdout_fraction_denominator,for_oodle_version_major);
#ifdef __OODLE2X_H_INCLUDED__
//OodleX_Shutdown();
OodleX_Shutdown(NULL,OodleX_Shutdown_LogLeaks_Yes,0);
#endif
//OodleXLog_Printf_v1("press a key\n");
//fgetc(stdin);
return ret;
}
//=================================================================
static void OodleLog_PacketCompression(OO_S64 tot_rawLen,OO_S64 tot_compLen,OO_S64 tot_numPackets)
{
OodleXLog_Printf_v0("%.1f -> %.1f average = %.3f:1 = %.2f%% reduction\n",
(double)tot_rawLen/tot_numPackets, (double)tot_compLen/tot_numPackets,
(double)tot_rawLen/tot_compLen,(double)(tot_rawLen-tot_compLen)*100.0/tot_rawLen);
OodleXLog_Printf_v1("%d packets; %lld -> %lld total bytes = %.2f%% of original\n",
(int)tot_numPackets,(long long)tot_rawLen,(long long)tot_compLen,(double)tot_compLen*100.0/tot_rawLen);
}
//=================================================================
struct PacketCompressResult
{
OO_S32 raw_size,compressed_size;
};
#if EXAMPLE_PACKET_DO_SORT
struct PacketCompressResult_Compare_raw_size
{
bool operator () (const PacketCompressResult & lhs,const PacketCompressResult & rhs) const
{
return lhs.raw_size < rhs.raw_size;
}
};
struct PacketCompressResult_Compare_compression_ratio
{
bool operator () (const PacketCompressResult & lhs,const PacketCompressResult & rhs) const
{
// multiply could overflow OO_S32 so bump it up :
return (OO_U64) lhs.compressed_size * rhs.raw_size < (OO_U64) rhs.compressed_size * lhs.raw_size;
}
};
static void OodleLog_PacketCompressResults(PacketCompressResult * results, OO_SINTa count)
{
OO_SINTa decile = count/10;
if ( decile == 0 ) return;
OodleXLog_Printf_v1("-------------------------------------------------------------\n");
for(int rep=0;rep<2;rep++)
{
if ( rep == 0 )
{
OodleXLog_Printf_v1("Compression by size decile; %d packets per decile\n",(int)decile);
// sort by raw size :
std::sort(results,results+count,PacketCompressResult_Compare_raw_size());
}
else
{
OodleXLog_Printf_v1("Compression by ratio decile; %d packets per decile\n",(int)decile);
// sort by raw size :
std::sort(results,results+count,PacketCompressResult_Compare_compression_ratio());
}
for(int d=0;d<10;d++)
{
PacketCompressResult * base = results + decile * d;
OO_SINTa decile_tot_raw_size = 0;
OO_SINTa decile_tot_compressed_size = 0;
for(OO_SINTa p=0;p<decile;p++)
{
decile_tot_raw_size += base[p].raw_size;
decile_tot_compressed_size += base[p].compressed_size;
}
OodleXLog_Printf_v1("%d : %6.1f -> %6.1f = %.2f%% of original = %.2f%% reduction\n",
d,
(double)decile_tot_raw_size/decile,
(double)decile_tot_compressed_size/decile,
(double)decile_tot_compressed_size*100.0/decile_tot_raw_size,
(double)(decile_tot_raw_size-decile_tot_compressed_size)*100.0/decile_tot_raw_size);
}
}
OodleXLog_Printf_v1("-------------------------------------------------------------\n");
}
#else
static void OodleLog_PacketCompressResults(PacketCompressResult * results, OO_SINTa count)
{
OOEX_UNUSED_VARIABLE(results);
OOEX_UNUSED_VARIABLE(count);
}
#endif // EXAMPLE_PACKET_DO_SORT
//=================================================================
TCP variants of the OodleNetwork1 tests
//=================================================================
TestOodleNetwork_SelectDictionaryAndTrain :
fills out on1_shared, on1_initial_state and on1_dic
uses the provided io_packet_train training data to both make the dictionary and
initialize the on1_initial_state
fills either tcp_initial_state or udp_initial_state or both. Set the one you don't need to NULL.
on1_shared and on1_dic are always filled.
static void TestOodleNetwork_SelectDictionaryAndTrain(
OodleNetwork1_Shared * on1_shared,
OodleNetwork1TCP_State * tcp_initial_state,
OodleNetwork1UDP_State * udp_initial_state2,
int on1_ht_bits,
void * on1_dic,
OO_SINTa on1_dic_size,
const OodleIOBuffer & iob_packet_train)
{
if ( ! OodleNetwork1_SelectDictionarySupported() )
{
OOEX_ASSERT_FAILURE_ALWAYS("Can't build dictionary on embedded platforms.");
return;
}
// parse the training set to vectors :
const OO_U8 * packet_bin_buf = iob_packet_train.ptr;
const OO_SINTa packet_bin_size = iob_packet_train.size;
vector<const void *> dictionary_packet_pointers;
vector<OO_S32> dictionary_packet_sizes;
dictionary_packet_pointers.reserve(4096);
dictionary_packet_sizes.reserve(4096);
vector<const void *> dictest_packet_pointers;
vector<OO_S32> dictest_packet_sizes;
dictest_packet_pointers.reserve(4096);
dictest_packet_sizes.reserve(4096);
vector<const void *> training_packet_pointers;
vector<OO_S32> training_packet_sizes;
training_packet_pointers.reserve(4096);
training_packet_sizes.reserve(4096);
// get the number of channels ; first dword in packet bin :
const OO_U8 * packet_bin_ptr = packet_bin_buf;
const OO_U8 * packet_bin_end = packet_bin_ptr + packet_bin_size;
OO_S32 num_channels = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
if ( num_channels == 0 ) num_channels = 1;
OOEX_UNUSED_VARIABLE(num_channels);
OO_SINTa training_tot_raw_len = 0;
OO_SINTa dictionary_tot_raw_len = 0;
OO_SINTa dictest_tot_raw_len = 0;
while( packet_bin_ptr < packet_bin_end )
{
// grab the current packet header :
OO_S32 channel = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OO_S32 bytes = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_ASSERT( channel >= 0 && channel < num_channels );
OOEX_ASSERT( bytes > 0 );
OOEX_UNUSED_VARIABLE(channel);
if ( packet_bin_ptr + bytes > packet_bin_end )
{
// partial end packet
break;
}
// put packet in one of three data sets :
// semi-random :
int select = irandmod(3);
if ( select == 0 )
{
training_packet_pointers.push_back( packet_bin_ptr );
training_packet_sizes.push_back( bytes );
training_tot_raw_len += bytes;
}
else if ( select == 1 )
{
dictionary_packet_pointers.push_back( packet_bin_ptr );
dictionary_packet_sizes.push_back( bytes );
dictionary_tot_raw_len += bytes;
}
else
{
dictest_packet_pointers.push_back( packet_bin_ptr );
dictest_packet_sizes.push_back( bytes );
dictest_tot_raw_len += bytes;
}
packet_bin_ptr += bytes;
}
OOEX_ASSERT( packet_bin_ptr == packet_bin_end );
// 64M limit for dictest to keep speed reasonable
while ( dictest_tot_raw_len > 64*1024*1024 )
{
// randomly move one from dictest to training :
int dictest_packet_pointers_size = dictest_packet_pointers.size32();
int i = irandmod(dictest_packet_pointers_size);
const void * ptr = dictest_packet_pointers[i];
OO_S32 size = dictest_packet_sizes[i];
training_packet_pointers.push_back(ptr);
training_packet_sizes.push_back(size);
training_tot_raw_len += size;
dictest_packet_pointers.erase_u(i);
dictest_packet_sizes.erase_u(i);
dictest_tot_raw_len -= size;
}
OO_S32 num_training_packets = training_packet_pointers.size32();
OO_S32 num_dictionary_packets = dictionary_packet_pointers.size32();
OO_S32 num_dictest_packets = dictest_packet_pointers.size32();
if ( on1_dic_size > dictionary_tot_raw_len )
{
OodleXLog_Printf_v1("WARNING : training data not big enough for dictionary, clamping.\n");
}
on1_dic_size = OOEX_MIN(on1_dic_size,dictionary_tot_raw_len);
OodleXLog_Printf_v1("clamped on1_dic_size = %d ; training len = %d\n",(int)on1_dic_size,(int)training_tot_raw_len);
#if EXAMPLE_PACKET_OODLENETWORK_DO_TRIALS
//OO_S32 num_trials = 10;
//OO_S32 num_generations = 2;
OO_S32 num_trials = 3;
OO_S32 num_generations = 1;
double randomness = 200.0;
#else
OO_S32 num_trials = 1;
OO_S32 num_generations = 0;
double randomness = 0.0;
#endif
// dic_file_name (sdfp_dic.bin) is a cache to speed up repeated runs of this function
// only used if EXAMPLE_PACKET_OODLENETWORK_LOAD_EXISTING is enabled
// it is *not* loaded by the runtime (that's the _runtimedata.bin)
// only loaded here
const char * dic_file_name = EXAMPLE_PACKET_DEFAULT_OUTPUT_PATH "sdfp_dic.bin";
// dic_file_name = NULL to not save
//const char * dic_file_name = NULL;
bool do_make_dic = true;
bool do_load_existing = EXAMPLE_PACKET_OODLENETWORK_LOAD_EXISTING != 0;
// On platforms where SelectDictionary funcs are not supported, force load existing
if ( ! OodleNetwork1_SelectDictionarySupported() )
{
do_load_existing = true;
}
if ( do_load_existing && dic_file_name != NULL )
{
OodleXLog_Printf_v1("Trying to load : %s\n",dic_file_name);
OO_SINTa dic_file_size;
void * dic_file_buf = read_whole_file(dic_file_name,&dic_file_size);
if ( dic_file_buf != NULL )
{
if ( on1_dic_size == (OO_SINTa)dic_file_size )
{
OodleXLog_Printf_v1("...got it!\n");
memcpy(on1_dic,dic_file_buf,on1_dic_size);
do_make_dic = false;
}
else
{
OodleXLog_Printf_v1("...wrong size!\n");
}
free(dic_file_buf);
}
else
{
OodleXLog_Printf_v1("...not found!\n");
}
}
if ( do_make_dic )
{
OodleNetwork1_SelectDictionaryFromPackets_Trials(on1_dic,(OO_S32)on1_dic_size,
on1_ht_bits,
dictionary_packet_pointers.data(),
dictionary_packet_sizes.data(),num_dictionary_packets,
dictest_packet_pointers.data(),
dictest_packet_sizes.data(),num_dictest_packets,
num_trials,randomness,num_generations);
//----------------------
// optionally write dictionary
// this isn't needed at runtime; the whole dictionary is stored in the "runtimedata" file as well
if ( dic_file_name )
{
OodleXLog_Printf_v1("writing dictionary to: %s\n",dic_file_name);
OO_BOOL write_ok = write_whole_file(dic_file_name,on1_dic,on1_dic_size);
OOEX_ASSERT_ALWAYS( write_ok );
}
}
if ( on1_shared )
OodleNetwork1_Shared_SetWindow(on1_shared,on1_ht_bits,on1_dic,(OO_S32)on1_dic_size);
// this is done by OodleNetwork1UDP_Train :
//OodleNetwork1UDP_State_Reset(on1_initial_state);
if ( tcp_initial_state )
{
OodleNetwork1TCP_Train(
tcp_initial_state,
on1_shared,
training_packet_pointers.data(),
training_packet_sizes.data(),
num_training_packets);
}
if ( udp_initial_state2 )
{
OodleNetwork1UDP_Train(
udp_initial_state2,
on1_shared,
training_packet_pointers.data(),
training_packet_sizes.data(),
num_training_packets);
}
}
TestOodleNetwork1TCPPacketCoder_Transmission :
Test coding of iob_packet_test
Use the previously trained on1_shared and on1_initial_state
static void TestOodleNetwork1TCPPacketCoder_Transmission(
const OodleNetwork1_Shared * on1_shared,
const OodleNetwork1TCP_State * on1_initial_state,
const OodleIOBuffer & iob_packet_test )
{
//===========================================
// point at the packet data :
const OO_U8 * packet_bin_buf = iob_packet_test.ptr;
OO_SINTa packet_bin_size = iob_packet_test.size;
// get the number of channels ; first dword in packet bin :
const OO_U8 * packet_bin_ptr = packet_bin_buf;
const OO_U8 * packet_bin_end = packet_bin_ptr + packet_bin_size;
OO_S32 num_channels = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
//===========================================
// initialize an OodleNetwork1 State for each channel :
vector<OodleNetwork1TCP_State *> encoders;
vector<OodleNetwork1TCP_State *> decoders;
encoders.resize(num_channels);
decoders.resize(num_channels);
for (int c=0;c<num_channels;c++)
{
encoders[c] = (OodleNetwork1TCP_State *) malloc( OodleNetwork1TCP_State_Size() );
decoders[c] = (OodleNetwork1TCP_State *) malloc( OodleNetwork1TCP_State_Size() );
OodleNetwork1TCP_State_InitAsCopy( encoders[c], on1_initial_state );
OodleNetwork1TCP_State_InitAsCopy( decoders[c], on1_initial_state );
}
//-------------------------------------------
vector<PacketCompressResult> results;
results.reserve(65536);
//-------------------------------------------
// make some buffers for compressed & decompressed data
// (we'll resize as needed)
vector<OO_U8> compv; compv.reserve(65536);
vector<OO_U8> decompv; decompv.reserve(65536);
OO_S64 tot_rawLen = 0;
OO_S64 tot_compLen = 0;
OO_S64 tot_numPackets = 0;
while( packet_bin_ptr < packet_bin_end )
{
// grab the current packet header :
OO_S32 channel = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OO_S32 bytes = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_ASSERT( channel >= 0 && channel < num_channels );
OOEX_ASSERT( bytes > 0 );
if ( packet_bin_ptr + bytes > packet_bin_end )
{
// partial end packet
break;
}
const OO_U8 * rawPtr = packet_bin_ptr;
OO_SINTa curLen = bytes;
compv.resize( OodleNetwork1_CompressedBufferSizeNeeded(curLen) );
decompv.resize( curLen + OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN );
OO_U8 * compPtr = compv.data();
OO_U8 * decomp = decompv.data();
OO_SINTa cur_compLen =
OodleNetwork1TCP_Encode(encoders[channel],on1_shared,rawPtr,curLen,compPtr);
// check I'm not lying about compLen :
compPtr[cur_compLen] = 0xCD;
//OO_SINTa dec_compLen =
OodleNetwork1TCP_Decode(decoders[channel],on1_shared,compPtr,cur_compLen,
decomp,bytes);
//OOEX_ASSERT_ALWAYS( cur_compLen == dec_compLen );
OOEX_ASSERT_ALWAYS( memcmp(packet_bin_ptr,decomp,bytes) == 0 );
packet_bin_ptr += bytes;
tot_rawLen += bytes;
tot_compLen += cur_compLen;
tot_numPackets ++;
results.push_back();
results.back().raw_size = (OO_S32) bytes;
results.back().compressed_size = (OO_S32) cur_compLen;
}
for (int c=0;c<num_channels;c++)
{
free(encoders[c]);
free(decoders[c]);
}
encoders.release();
decoders.release();
OodleLog_PacketCompression(tot_rawLen,tot_compLen,tot_numPackets);
OodleLog_PacketCompressResults(&results[0],results.size());
}
TestOodleNetwork1TCPPacketCoder
Test OodleNetwork1 on the packet data
does training, then compresses with the trained data
static void TestOodleNetwork1TCPPacketCoder(
const OodleIOBuffer & iob_packet_train ,
const OodleIOBuffer & iob_packet_test ,
int on1_dic_mb , int on1_ht_bits)
{
OO_SINTa on1_dic_size = on1_dic_mb*1024*1024;
on1_dic_size = OOEX_CLAMP(on1_dic_size,1024*1024,OODLENETWORK1_MAX_DICTIONARY_SIZE);
OodleXLog_Printf_v1("OodleNetwork1 TCP models take : %d bytes per channel, %d bytes shared\n",
(int) OodleNetwork1TCP_State_Size(),
(int)(OodleNetwork1_Shared_Size(on1_ht_bits) + on1_dic_size));
//-------------------------------------
// OodleNetwork1 training requires a dictionary (on1_dic) of bytes to reference
// The "on1_shared" object is static and shared by all channels
// it is built once from the on1_dic
// The "on1_initial_state" is a trained initial state
// this state is copied into the per-channel states
void * on1_dic = malloc( on1_dic_size );
OodleNetwork1_Shared * on1_shared = (OodleNetwork1_Shared *) malloc( OodleNetwork1_Shared_Size(on1_ht_bits) );
OOEX_ASSERT_ALWAYS( on1_shared != NULL );
OodleNetwork1TCP_State * on1_initial_state = (OodleNetwork1TCP_State *) malloc( OodleNetwork1TCP_State_Size() );
OOEX_ASSERT_ALWAYS( on1_initial_state != NULL );
TestOodleNetwork_SelectDictionaryAndTrain( on1_shared,
on1_initial_state,NULL,
on1_ht_bits, on1_dic, on1_dic_size, iob_packet_train );
//-----------------------------------------------------
OodleXLog_Printf_v1("OodleNetwork1 TCP [%d|%d] : ",on1_dic_mb,on1_ht_bits);
TestOodleNetwork1TCPPacketCoder_Transmission( on1_shared, on1_initial_state, iob_packet_test );
//-----------------------------------------------------
free(on1_dic);
free(on1_shared);
free(on1_initial_state);
}
//=============================================================================
UDP variants of the OodleNetwork1 tests
//=============================================================================
TestOodleNetwork1UDPPacketCoder_Transmission :
Test coding of iob_packet_test
Use the previously trained on1_shared and on1_initial_state
static void TestOodleNetwork1UDPPacketCoder_Transmission(
const OodleNetwork1_Shared * on1_shared,
const OodleNetwork1UDP_State * state,
const OodleIOBuffer & iob_packet_test )
{
//-------------------------------------------
vector<PacketCompressResult> results;
results.reserve(65536);
//-------------------------------------------
// make some buffers for compressed & decompressed data
// (we'll resize as needed)
OO_S64 tot_rawLen = 0;
OO_S64 tot_compLen = 0;
OO_S64 tot_numPackets = 0;
// scratch space for just one packet compressed & decompressed
vector<OO_U8> compv; compv.reserve(65536);
vector<OO_U8> decompv; decompv.reserve(65536);
//===========================================
// point at the packet data :
const OO_U8 * packet_bin_buf = iob_packet_test.ptr;
OO_SINTa packet_bin_size = iob_packet_test.size;
// get the number of channels ; first dword in packet bin :
const OO_U8 * packet_bin_ptr = packet_bin_buf;
const OO_U8 * packet_bin_end = packet_bin_ptr + packet_bin_size;
OO_S32 num_channels = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
if ( num_channels == 0 ) num_channels = 1;
OOEX_UNUSED_VARIABLE(num_channels);
//===========================================
while( packet_bin_ptr < packet_bin_end )
{
// grab the current packet header :
OO_S32 channel = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_UNUSED_VARIABLE(channel);
OO_S32 bytes = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_ASSERT( channel >= 0 && channel < num_channels );
OOEX_ASSERT( bytes > 0 );
if ( packet_bin_ptr + bytes > packet_bin_end )
{
// partial end packet
break;
}
const OO_U8 * rawPtr = packet_bin_ptr;
OO_SINTa curLen = bytes;
OO_SINTa comp_needed = OodleNetwork1_CompressedBufferSizeNeeded(curLen);
if ( compv.size32() < comp_needed )
{
compv.resize(comp_needed);
}
// decomp must be sized to at least curLen + OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN
if ( decompv.size32() < curLen+OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN )
decompv.resize(curLen+OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN);
OO_U8 * comp = compv.data();
OO_U8 * decomp = decompv.data();
OO_SINTa cur_compLen =
OodleNetwork1UDP_Encode(state,on1_shared,rawPtr,curLen,comp);
// check I'm not lying about compLen :
comp[cur_compLen] = 0xCD;
//OO_SINTa dec_compLen =
OO_BOOL ok = OodleNetwork1UDP_Decode(state,on1_shared,comp,cur_compLen,
decomp,bytes);
OOEX_ASSERT_ALWAYS( ok );
//OOEX_ASSERT_ALWAYS( cur_compLen == dec_compLen );
OOEX_ASSERT_ALWAYS( memcmp(packet_bin_ptr,decomp,bytes) == 0 );
packet_bin_ptr += bytes;
tot_rawLen += bytes;
tot_compLen += cur_compLen;
tot_numPackets ++;
results.push_back();
results.back().raw_size = (OO_S32) bytes;
results.back().compressed_size = (OO_S32) cur_compLen;
}
OodleLog_PacketCompression(tot_rawLen,tot_compLen,tot_numPackets);
OodleLog_PacketCompressResults(&results[0],results.size());
}
TestOodleNetwork1UDPPacketCoder
Train the UDP coder then test using it
This function also demonstrated how you would compress the model and save it to disk
In real runtimes, you would just start from loading the compressed model
-----------------------------
OodleNetwork1_SavedModel_Header is an example header for the saved Oodle Network trained model.
It is not intended that you use this literally in your game. You should write your own IO code using
your engine's IO systems to persist the network model.
Of note : the oodle_major_version is stored in this example header. If that is checked, then it
forces the saved state to be regenerated whenever Oodle's version number is rev'ed. That is usually
not necessary, old states will still work with most revs. So you may remove the check of
oodle_major_version if you like.
NOTE : in real production code, your saved model should probably have encryption and/or a
checksum applied for integrity checking and to prevent tampering.
struct OodleNetwork1_SavedModel_Header
{
#define ON1_MAGIC 0x11235801
OO_U32 magic;
OO_U32 compressor;
OO_U32 ht_bits;
OO_U32 dic_size;
OO_U32 oodle_major_version;
OO_U32 dic_complen;
OO_U32 statecompacted_size;
OO_U32 statecompacted_complen;
};
OodleNetwork1_SavedModel_Header is written to file like a flat struct,
but we ensure it's always little endian.
static void OodleNetwork1_SavedModel_Header_Read(OodleNetwork1_SavedModel_Header * pHeader,const void * from_memory)
{
const OO_U32 * from_ptr = (const OO_U32 *)from_memory;
pHeader->magic = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->compressor = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->ht_bits = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->dic_size = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->oodle_major_version = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->dic_complen = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->statecompacted_size = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->statecompacted_complen = OOEX_GET32_LE(from_ptr); from_ptr++;
OOEX_ASSERT( (((OO_U8 *)from_ptr) - ((OO_U8 *)from_memory)) == sizeof(OodleNetwork1_SavedModel_Header) );
}
static void OodleNetwork1_SavedModel_Header_Write(const OodleNetwork1_SavedModel_Header * pHeader,void * to_memory)
{
OO_U32 * to_ptr = (OO_U32 *)to_memory;
OOEX_PUT32_LE(to_ptr,pHeader->magic); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->compressor); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->ht_bits); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->dic_size); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->oodle_major_version); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->dic_complen); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->statecompacted_size); to_ptr++;
OOEX_PUT32_LE(to_ptr,pHeader->statecompacted_complen); to_ptr++;
OOEX_ASSERT( (((OO_U8 *)to_ptr) - ((OO_U8 *)to_memory)) == sizeof(OodleNetwork1_SavedModel_Header) );
}
struct OodleNetwork1_Compressor
{
void * dic;
OodleNetwork1UDP_State * state;
OodleNetwork1_Shared * shared;
};
static void OodleNetwork1_Compressor_Free(OodleNetwork1_Compressor * pCompressor)
{
free(pCompressor->dic);
free(pCompressor->shared);
free(pCompressor->state);
}
static bool OodleNetwork1_Compressor_LoadFromFileData( OodleNetwork1_Compressor * pCompressor, const void * fileData, OO_SINTa fileSize )
{
// parse header :
OodleNetwork1_SavedModel_Header header;
OodleNetwork1_SavedModel_Header_Read(&header,fileData);
if ( header.magic != ON1_MAGIC )
{
OodleXLog_Printf_v0("runtimedata_file ON1_MAGIC mismatch\n");
return false;
}
// optional check for oodle_major_version being the same
// NOTE : this is more conservative than necessary
// you may disable this check to keep loading old runtimedata files
#if 0
if ( header.oodle_major_version < 1 )
{
OodleXLog_Printf_v0("runtimedata_file version too old!\n");
return false;
}
if ( header.oodle_major_version > OODLE2NET_VERSION_MAJOR )
{
OodleXLog_Printf_v0("runtimedata_file version newer that SDK!\n");
return false;
}
#endif
OO_S32 on1udpnew_ht_bits = header.ht_bits;
OO_SINTa on1udpnew_dic_size = header.dic_size;
OO_SINTa on1udpnew_dic_complen = header.dic_complen;
OO_SINTa on1udpnew_statecompacted_size = header.statecompacted_size;
OO_SINTa on1udpnew_statecompacted_complen = header.statecompacted_complen;
// sanity checks on the header :
// (in real use these should be if's and the header should be protected to
// forbid tampering)
OOEX_ASSERT_ALWAYS( on1udpnew_dic_size >= on1udpnew_dic_complen );
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size+5 >= on1udpnew_statecompacted_complen );
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size > 0 && on1udpnew_statecompacted_size < OodleNetwork1UDP_StateCompacted_MaxSize() );
OOEX_ASSERT_ALWAYS( fileSize == (OO_S64)sizeof(OodleNetwork1_SavedModel_Header) + on1udpnew_dic_complen + on1udpnew_statecompacted_complen );
OodleXLog_Printf_v1("OodleNetwork1UDP Loading; dic comp %d , state %d->%d\n",
(int)on1udpnew_dic_complen,(int)on1udpnew_statecompacted_size,(int)on1udpnew_statecompacted_complen);
//-------------------------------------------
// decompress on1udpnew_dic and on1udpnew_statecompacted
pCompressor->dic = malloc(on1udpnew_dic_size);
const void * on1udpnew_dic_comp_ptr = (const OO_U8 *)(fileData) + sizeof(OodleNetwork1_SavedModel_Header);
OodleNetwork1UDP_StateCompacted * on1udpnew_compacted = (OodleNetwork1UDP_StateCompacted *) malloc( on1udpnew_statecompacted_size );
void * on1udpnew_statecompacted_comp_ptr = (OO_U8 *)on1udpnew_dic_comp_ptr + on1udpnew_dic_complen;
if ( header.compressor == (OO_U32)OodleLZ_Compressor_Invalid )
{
OOEX_ASSERT_ALWAYS( on1udpnew_dic_complen == on1udpnew_dic_size );
memcpy(pCompressor->dic,on1udpnew_dic_comp_ptr,on1udpnew_dic_size);
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size == on1udpnew_statecompacted_complen );
memcpy(on1udpnew_compacted,on1udpnew_statecompacted_comp_ptr,on1udpnew_statecompacted_size);
}
else
{
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
OO_SINTa decomp_dic_size = OodleLZ_Decompress(on1udpnew_dic_comp_ptr,on1udpnew_dic_complen,pCompressor->dic,on1udpnew_dic_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( decomp_dic_size == on1udpnew_dic_size );
OO_SINTa decomp_statecompacted_size = OodleLZ_Decompress(on1udpnew_statecompacted_comp_ptr,on1udpnew_statecompacted_complen,on1udpnew_compacted,on1udpnew_statecompacted_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( decomp_statecompacted_size == on1udpnew_statecompacted_size );
#else
OOEX_ASSERT_FAILURE_ALWAYS("no USE_OODLE_LZ_DATA_COMPRESSION");
#endif
}
//----------------------------------------------
// Uncompact the "Compacted" state into a usable state
OO_SINTa on1udpnew_state_size = OodleNetwork1UDP_State_Size();
pCompressor->state = (OodleNetwork1UDP_State *) malloc( on1udpnew_state_size );
OOEX_ASSERT_ALWAYS( pCompressor->state != NULL );
if ( ! OodleNetwork1UDP_State_Uncompact_ForVersion(pCompressor->state,on1udpnew_compacted,header.oodle_major_version) )
{
OodleXLog_Printf_v0("OodleNetwork1UDP_State_Uncompact failed\n");
return false;
}
free(on1udpnew_compacted);
//----------------------------------------------
// fill out on1udpnew_shared from the dictionary
OO_SINTa shared_size = OodleNetwork1_Shared_Size(header.ht_bits);
pCompressor->shared = (OodleNetwork1_Shared *) malloc( shared_size );
OOEX_ASSERT_ALWAYS( pCompressor->shared != NULL );
OodleNetwork1_Shared_SetWindow(pCompressor->shared,on1udpnew_ht_bits,pCompressor->dic,(OO_S32)on1udpnew_dic_size);
return true;
}
static bool OodleNetwork1_Compressor_WriteToFile(const char * runtimedata_fileName,
const void * on1udpnew_dic,
OO_SINTa on1udpnew_dic_size,
const OodleNetwork1UDP_State *on1udpnew_state,
OO_S32 on1udpnew_ht_bits,
OodleLZ_Compressor file_compressor,
int for_oodle_version_major)
{
//-----------------------------------------------------
// for saving the state to file,
// convert it to a "Compacted" state
// this is a smaller, non-runtime representation of the state
OO_SINTa on1udpnew_statecompacted_maxsize = OodleNetwork1UDP_StateCompacted_MaxSize();
OodleNetwork1UDP_StateCompacted * on1udpnew_compacted = (OodleNetwork1UDP_StateCompacted *) malloc( on1udpnew_statecompacted_maxsize );
OO_SINTa on1udpnew_statecompacted_size = OodleNetwork1UDP_State_Compact_ForVersion(on1udpnew_compacted,on1udpnew_state,for_oodle_version_major);
OO_SINTa on1udpnew_state_size = OodleNetwork1UDP_State_Size();
OodleXLog_Printf_v1("OodleNetwork1UDP models compacted : %d bytes (from %d)\n",
(int)on1udpnew_statecompacted_size,
(int)on1udpnew_state_size);
//-----------------------------------------------------
// compress the block we will save
// we need to save on1udpnew_dic and on1udpnew_compacted
// we can make the size of the saved state file even smaller by
// running normal OodleLZ compress on it
OO_SINTa compressed_buffer_alloc_size = sizeof(OodleNetwork1_SavedModel_Header);
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
compressed_buffer_alloc_size += OodleLZ_GetCompressedBufferSizeNeeded(file_compressor, on1udpnew_dic_size + on1udpnew_statecompacted_size );
#else
compressed_buffer_alloc_size += on1udpnew_dic_size + on1udpnew_statecompacted_size;
#endif
void * compressed_buffer = malloc( compressed_buffer_alloc_size );
OO_U8 * comp_ptr = (OO_U8 *) compressed_buffer;
// reserve space for header
OO_U8 * header_pointer = comp_ptr;
comp_ptr += sizeof(OodleNetwork1_SavedModel_Header);
OO_SINTa on1udpnew_dic_complen;
if (file_compressor == OodleLZ_Compressor_Invalid )
{
on1udpnew_dic_complen = on1udpnew_dic_size;
memcpy(comp_ptr,on1udpnew_dic,on1udpnew_dic_size);
}
else
{
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
// For real production you should use OodleLZ_CompressionLevel_Optimal2
// for maximum compression without affecting decode speed
// (just slower to encode)
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Optimal2;
on1udpnew_dic_complen = OodleXLZ_Compress_AsyncAndWait(OodleXAsyncSelect_All,file_compressor,on1udpnew_dic,on1udpnew_dic_size,comp_ptr,level);
OOEX_ASSERT_ALWAYS( on1udpnew_dic_complen > 0 );
#else
on1udpnew_dic_complen = -1;
OOEX_ASSERT_FAILURE_ALWAYS("no USE_OODLE_LZ_DATA_COMPRESSION");
#endif
}
comp_ptr += on1udpnew_dic_complen;
OO_SINTa on1udpnew_statecompacted_complen;
if ( file_compressor == OodleLZ_Compressor_Invalid )
{
memcpy(comp_ptr,on1udpnew_compacted,on1udpnew_statecompacted_size);
on1udpnew_statecompacted_complen = on1udpnew_statecompacted_size;
}
else
{
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
on1udpnew_statecompacted_complen = OodleLZ_Compress(file_compressor,on1udpnew_compacted,on1udpnew_statecompacted_size,comp_ptr,OodleLZ_CompressionLevel_Fast);
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_complen > 0 );
#else
OOEX_ASSERT_FAILURE_ALWAYS("no USE_OODLE_LZ_DATA_COMPRESSION");
on1udpnew_statecompacted_complen = -1;
#endif
}
comp_ptr += on1udpnew_statecompacted_complen;
OO_SINTa total_complen = (OO_SINTa) ( comp_ptr - (OO_U8 *) compressed_buffer );
OodleXLog_Printf_v1("OodleNetwork1UDP runtimedata file size : %d\n",(int)total_complen);
free(on1udpnew_compacted); on1udpnew_compacted = NULL;
//-----------------------------------------------------
// fill the header :
// note : you should make a more robust & portable header for real use!
OodleNetwork1_SavedModel_Header header = { 0 };
header.magic = ON1_MAGIC;
header.ht_bits = on1udpnew_ht_bits;
header.compressor = (OO_U32) file_compressor;
header.dic_size = (OO_U32) on1udpnew_dic_size;
header.oodle_major_version = for_oodle_version_major;
header.dic_complen = (OO_U32) on1udpnew_dic_complen;
header.statecompacted_size = (OO_U32) on1udpnew_statecompacted_size;
header.statecompacted_complen = (OO_U32) on1udpnew_statecompacted_complen;
OodleNetwork1_SavedModel_Header_Write(&header,header_pointer);
//-----------------------------------------------------
// write the compressed buffer containing on1udpnew_dic and on1udpnew_state to a file
OodleXLog_Printf_v1("Writing runtimedata : %s\n",runtimedata_fileName);
OO_BOOL write_ok = write_whole_file(runtimedata_fileName,compressed_buffer,total_complen);
if ( ! write_ok )
{
OodleXLog_Printf_v1("ERROR : failed to write : %s\n",runtimedata_fileName);
return false;
}
return true;
}
static void TestOodleNetwork1UDPPacketCoder(
const OodleIOBuffer & iob_packet_train ,
const OodleIOBuffer & iob_packet_test ,
int on1udpnew_dic_mb , int on1udpnew_ht_bits,
int for_oodle_version_major)
{
OO_SINTa on1udpnew_dic_size = on1udpnew_dic_mb*1024*1024;
on1udpnew_dic_size = OOEX_CLAMP(on1udpnew_dic_size,1024*1024,OODLENETWORK1_MAX_DICTIONARY_SIZE);
OO_SINTa shared_size = OodleNetwork1_Shared_Size(on1udpnew_ht_bits);
OO_SINTa on1udpnew_state_size = OodleNetwork1UDP_State_Size();
OodleXLog_Printf_v1("OodleNetwork1 UDP models take : %d bytes shared (%d + %d + %d)\n",
(int) (on1udpnew_state_size + shared_size + on1udpnew_dic_size),
(int)on1udpnew_state_size,(int) shared_size,(int) on1udpnew_dic_size
);
// this is all the data loaded by the runtime
// (see example_network_client.cpp)
const char * runtimedata_fileName = EXAMPLE_PACKET_DEFAULT_OUTPUT_PATH "example_packet_on1udpnew_runtimedata.bin";
//-------------------------------------
// train the dictionary and on1udpnew state
// this is done offline and the result is saved to a file
#if EXAMPLE_PACKET_OODLENETWORK_LOAD_EXISTING
// skip training and just load the file we made last time
OOEX_UNUSED_VARIABLE(iob_packet_train);
#else
if ( 1 )
{
// OodleNetwork1UDP training requires a dictionary (on1udpnew_dic) of bytes to reference
// The "on1udpnew_shared" object is static and shared by all channels
// it is built once from the on1udpnew_dic
// The "on1udpnew_initial_state" is a trained initial state
// this state is copied into the per-channel states
void * on1udpnew_dic = malloc( on1udpnew_dic_size );
OodleNetwork1_Shared * on1udpnew_shared = (OodleNetwork1_Shared *) malloc( shared_size );
OOEX_ASSERT_ALWAYS( on1udpnew_shared != NULL );
OodleNetwork1UDP_State * on1udpnew_state = (OodleNetwork1UDP_State *) malloc( on1udpnew_state_size );
OOEX_ASSERT_ALWAYS( on1udpnew_state != NULL );
TestOodleNetwork_SelectDictionaryAndTrain(
on1udpnew_shared,
NULL,on1udpnew_state,
on1udpnew_ht_bits, on1udpnew_dic, on1udpnew_dic_size, iob_packet_train );
// on1udpnew_shared is NOT written, it must be remade on load
free(on1udpnew_shared); on1udpnew_shared = NULL;
//-----------------------------------------------------
OodleXLog_Printf_v1("Saving state...\n");
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
// write the state compressed :
OodleLZ_Compressor file_compressor = OodleLZ_Compressor_Kraken;
OOEX_ASSERT( OodleLZ_Compressor_CanDecodeFuzzSafe(file_compressor) );
#else
// uncompressed state :
OodleLZ_Compressor file_compressor = OodleLZ_Compressor_Invalid;
#endif
bool ok =OodleNetwork1_Compressor_WriteToFile(runtimedata_fileName,on1udpnew_dic,on1udpnew_dic_size,on1udpnew_state,on1udpnew_ht_bits,file_compressor,for_oodle_version_major);
free(on1udpnew_dic);
free(on1udpnew_state);
if (! ok )
{
OodleXLog_Printf_v0("runtimedata save failed\n");
OOEX_ASSERT_FAILURE_ALWAYS("fail");
}
}
#endif // EXAMPLE_PACKET_OODLENETWORK_LOAD_EXISTING
// real runtime would start here
// trained model is already compressed and saved to disk
//-----------------------------------------------------
// load the saved runtimedata and use it for compression
if ( 1 )
{
// read the compressed runtimedata file
OO_SINTa runtimedata_fileSizeA = 0;
void * runtimedata_fileData = read_whole_file(runtimedata_fileName,&runtimedata_fileSizeA);
OOEX_ASSERT_ALWAYS( runtimedata_fileData != NULL );
if ( runtimedata_fileSizeA < 4096 )
{
OodleXLog_Printf_v0("runtimedata_fileSize too small\n");
OOEX_ASSERT_FAILURE_ALWAYS("fail");
}
OodleNetwork1_Compressor compressor = { 0 };
bool ok = OodleNetwork1_Compressor_LoadFromFileData(&compressor,runtimedata_fileData,runtimedata_fileSizeA);
free(runtimedata_fileData); runtimedata_fileData = NULL;
if (! ok )
{
OodleXLog_Printf_v0("runtimedata load failed\n");
OOEX_ASSERT_FAILURE_ALWAYS("fail");
}
//-----------------------------------------------------
// our runtime data structures are now loaded
// test compressing some packets
OodleXLog_Printf_v0("OodleNetwork1 UDP [%d|%d] : ",on1udpnew_dic_mb,on1udpnew_ht_bits);
TestOodleNetwork1UDPPacketCoder_Transmission( compressor.shared, compressor.state, iob_packet_test );
//-----------------------------------------------------
OodleNetwork1_Compressor_Free(&compressor);
}
//-----------------------------------------------------
}
//=================================================
A simple random number generator :
Marsaglia's KISS99
static OO_U32 x = 123456789,y = 362436000,z = 521288629,c = 7654321; // seeds
static OO_U32 KISS99()
{
x = 69069*x+12345;
y ^= (y<<13);
y ^= (y>>17);
y ^= (y<<5);
static const OO_U64 a = 698769069ULL;
OO_U64 t = a*z+c;
c = (OO_U32)(t>>32);
z = (OO_U32)t;
return (x+y+z);
}
// returns a value in [0,size-1]
static OO_U32 irandmod(OO_U32 size)
{
// use mul hi
return (OO_U32)((KISS99() * (OO_U64)size) >> 32);
}
Oodle Ext provides threading and IO helpers to run compression tasks multi-threaded. You don't need
to use them, you can just call the simple single threaded functions.
Functions in Oodle Ext start with OodleX_.
OodleX requires a single Init and Shutdown call per application run.
See Oodle2 Core vs Oodle2 Ext
Oodle async operations are coordinated through the OodleHandle
(see About OodleXHandle) which provides a way to check on async
work or IO's, or groups or chains of work and IO's.
It is recommended that you ship your game using Oodle Core only. Oodle Ext is intended for tools and
to get a quick start using Oodle. Some console & mobile platforms versions of Oodle do not include Oodle Ext.
Naming Conventions
OodleX tries to make it clear what its functions do through a few naming conventions.
Functions with _Async on the name start an async operation. Generally these functions return
immediately, but the actual operation is not done yet. Generally they return an OodleHandle.
Functions with _Wait on the name wait on an async operation. They may block a long time, depending
on the async operation. Typically they also free the operation or provide a flag for whether to free it.
Generally for each _Async function you want to pair it with a _Wait. There are function-specific
pairings, and those are generally indicated by a shared prefix in the function name, such as
OodleXLZ_Compress_Async then OodleXLZ_Compress_Wait_GetResult. If there is not a specific pair you can
always use OodleX_Wait on any OodleHandle.
Functions with _AsyncAndWait in the name indicate that they fire off an async op and then wait on it.
This is different than a synchronous function (sometimes indicated with _Sync in the name, but more often
just any function that doesn't have Async in the name); a synchronous
function runs on the calling thread, whereas an _AsyncAndWait function is fired off to some async runner
and then blocked on.
Some functions have Wide or Narrow in the name. Narrow means the task is run asynchronously, but is run
as a single non-separable operation. Wide means the task can be broken into several smaller pieces and the pieces
can be run simultaneously using several threads. These correspond to flags of OodleXAsyncSelect.
Functions generally have subsystem prefixes to indicate what kinds of data they work on. For example OodleWork_
prefix functions can only work on OodleHandles that correspond to Work objects. It's up to you to
keep handles in the correct subsytem. The generic Oodle_ prefix functions
(like OodleX_WaitAll or OodleX_GetStatus) work on any subsystem.
There are lots of enum arguments in Oodle. The values that the enums can take always start with the type
of the enum; eg. OodleXCopyFileFlags has values like OodleXCopyFileFlags_Overwrite and
OodleXCopyFileFlags_DontOverwriteExisting. One special common case is enums that describe a boolean
choice; in that case the name is always the enum type + _Yes or _No , as in OodleXHandleDeleteIfDone
which has values OodleXHandleDeleteIfDone_Yes and OodleXHandleDeleteIfDone_No.
Discussion
Get Info about a file ; if the file is not open yet, wait for itParameters
Return Value
| return | true if successful; if GetInfo returns false, pInto and pAlignmentRequired are untouched.
|
Discussion
This function is like OodleXIOQ_GetInfo , but will not return false if the Open operation is still pending; instead it
will block the calling thread until the Open is done so that info is available.
| OODLE_MALLOC_MINIMUM_ALIGNMENT |
|
|
#if defined(__RAD64__)
#define OODLE_MALLOC_MINIMUM_ALIGNMENT 16
#else
#define OODLE_MALLOC_MINIMUM_ALIGNMENT 8
#endif
Description
OodleMallocAligned will be asked to provide at least OODLE_MALLOC_MINIMUM_ALIGNMENT
OODLE_MALLOC_MINIMUM_ALIGNMENT is the size of two pointers.
| FAQ: How do I limit the encoder memory use? |
|
|
If you wish to limit the encoder memory use, for runtime encoding :
First of all use a compression level in the "fast/normal" range, not an optimal. The optimal levels
use a lot of memory, will call out to your installed malloc plugin (not use scratch), and is hard to
limit. The optimal levels are not intended for runtime compression.
You can pass in preallocated scratch memory to OodleLZ_Compress using scratchMem and scratchSize
arguments. The encoder will take memory from this pool first before calling to your installed allocator.
You can install your own allocator with OodleCore_Plugins_SetAllocators , and then log the allocations
or make it an error if OodleLZ_Compress tries to use the allocator. That way you can ensure that
OodleLZ_Compress only takes memory from the pool you pass in.
Okay, so now how much scratch memory to pass in?
It depends on the OodleLZ_CompressOptions and OodleLZ_CompressionLevel and OodleLZ_Compressor, as
well as the length of the buffer you compress.
You can call OodleLZ_GetCompressScratchMemBound to get a bound on how much memory will be used. If you
provide that much scratch, Oodle will not need to allocate.
OodleLZ_GetCompressScratchMemBound may return OODLELZ_SCRATCH_MEM_NO_BOUND which means it cannot provide
a strict bound.
By default, memory use grows roughly proportionally to buffer size. If you want to force a lower maximum
memory use, you can modify the OodleLZ_CompressOptions:matchTableSizeLog2
setting. The majority of scratch memory use comes from 4 bytes per matchTableSizeLog2 ; eg if
you set matchTableSizeLog2 = 20 , that would use 4 MB for the "match table".
Other than the match table, about 1 MB of additional scratch is needed by the encoder (for unbounded length
input buffers; smaller inputs need less), so at matchTableSizeLog2 = 20 , you would need about 5 MB,
4 MB from the match table and 1 MB for other.
Discussion
Get Info about a fileParameters
Return Value
| return | true if successful; if GetInfo returns false, pInto and pAlignmentRequired are untouched.
|
Discussion
If the file is not yet open, GetInfo will fail and return false. eg. if OodleXIOQ_OpenForRead_Async has been done
but the request is still pending.
If the file size can not be queried it is set to OODLEX_FILE_SIZE_INVALID.
If pAlignmentRequired is given, it is filled with the alignment required to use this file.
OODLEX_IO_MAX_ALIGNMENT is guaranteed to always be okay, so if you align to that then you are fine.
See About OodleIOQ for more about alignment.
| example_lz_threadphased : Example of 2-thread ThreadPhased decoding |
|
|
Discussion
Oodle example_lz_threadphased
This example demonstrates the ability of Kraken to decode using "ThreadPhased" parallelism
See About OodleLZ ThreadPhased Decode for details.
ThreadPhased decoding provides a 1X-2X speedup, typically around 33%-50%
This example implements an entire ThreadPhased decoder, in example_lz_threadphased_decompress.
The idea is you may take this example code and modify it to run in your own threading or job system,
so that you may implement the threading however you look.
example_lz_threadphased_decompress can be used without OodleX or the Oodle Worker system.
In this example I use OodleX_CreateThread and OodleX_Semaphore ; these are just handy cross-platform
implementations for me. The intention is that you replace them with your own threading system calls.
For the semaphore, it's important that it tries to avoid going into an OS Wait (thread sleep) when the
two threads are nearly synchronized. To avoid this you want a user-space spin-backoff loop to try to keep
the two threads awake together. One option is to use something like "fastsemaphore" as a wrapper to your
underlying OS semaphore :
http://cbloomrants.blogspot.com/2011/12/12-08-11-some-semaphores.html
If you use that "fastsemaphore" make sure to set
int spin_count = 1; // ! set this for your system
something like 100 is usually reasonable, and probably add a backoff.
#include "../include/oodle2x.h"
#include "ooex.h"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "make_example_input.h"
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz_threadphased
#endif
//===========================================================
// file names :
static const char * in_name = "oodle_example_input_file";
//===========================================================
static bool example_lz_threadphased_decompress(void * comp_buf,OO_SINTa comp_len,void * dec_buf,OO_SINTa dec_size,bool async);
//===========================================================
extern "C" int main(int argc,char *argv[])
{
// Init Oodle systems with default options :
if ( ! OodleX_Init_Default(OODLE_HEADER_VERSION) )
{
fprintf(stderr,"OodleX_Init failed.\n");
return 10;
}
if ( argc >= 2 ) in_name = argv[1];
else make_example_input(in_name);
OodleXLog_Printf_v1("example_lz_threadphased : %s\n",in_name);
// read the input file to the global buffer :
OO_S64 in_size_64;
void * in_buffer = OodleXIOQ_ReadMallocWholeFile_AsyncAndWait(in_name,&in_size_64);
if ( ! in_buffer)
{
OodleXLog_Printf_v0("failed to read %s\n",in_name);
return 10;
}
OO_SINTa in_size = OodleX_S64_to_SINTa_check( in_size_64 );
//=========================================
// select options :
OodleLZ_Compressor compressor = OodleLZ_Compressor_Kraken;
//OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Normal;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Fast;
// this example is only valid for compressors that support ThreadPhased decode : (eg. Kraken)
OOEX_ASSERT( OodleLZ_Compressor_CanDecodeThreadPhased(compressor) );
//=========================================
// allocate memory big enough for compressed data :
void * comp_buf = OodleXMalloc( OodleLZ_GetCompressedBufferSizeNeeded(compressor,in_size) );
// memory to decode to :
void * dec_buf = OodleXMalloc( in_size );
//=========================================
// compress the input :
// compress :
// this is just a normal whole-block compress, no special parallel mode is needed
OO_SINTa comp_len = OodleLZ_Compress(compressor,in_buffer,in_size,comp_buf,level);
OodleXLog_Printf_v1("Compressed : %d -> %d\n",(int)in_size,(int)comp_len);
//=======================================
We can decompress asynchronously using the "ThreadPhased" helper in OodleX.
The Narrow helper uses 2 threads and frees up the calling thread to do other work.
OodleXLZ_Decompress_ThreadPhased_Narrow_Async uses the OodleX Worker thread system, which is started by
default in OodleX_Init.
NOTE for maximum speed you should pass in the scratch space needed by OodleXLZ_Decompress_ThreadPhased_Narrow_Async
pre-allocated, so it doesn't have to do the allocation internally.
OodleXLog_Printf_v1("OodleXLZ_Decompress_ThreadPhased_Narrow_Async...\n");
OodleXHandle decomp_handle = OodleXLZ_Decompress_ThreadPhased_Narrow_Async(comp_buf,comp_len,dec_buf,in_size);
// .. can do other work on the main thread now ..
OodleXStatus decomp_status = OodleX_WaitAndDelete(decomp_handle);
if ( decomp_status == OodleXStatus_Error )
{
OodleXLog_Printf_v1("Error!\n");
}
// check it :
OOEX_ASSERT( memcmp(in_buffer,dec_buf,in_size) == 0 );
//=======================================
// do our own thread-phased decode :
OodleXLog_Printf_v1("example_lz_threadphased_decompress ");
// run a few reps to stress test :
for(int rep=0;rep<10;rep++)
{
// run async and sync options :
for(int async = 0;async<=1;async++)
{
OodleXLog_Printf_v1(async ? "+" : "-");
// wipe out dec_buf to make sure we decode correctly :
memset(dec_buf,0xEE,in_size);
if ( ! example_lz_threadphased_decompress(comp_buf,comp_len,dec_buf,in_size,!!async) )
{
OodleXLog_Printf_v1("Error!\n");
}
// check it :
OOEX_ASSERT( memcmp(in_buffer,dec_buf,in_size) == 0 );
}
}
OodleXLog_Printf_v1("\n");
//=======================================
OodleXFree(comp_buf);
OodleXFree(dec_buf);
OodleXFree_IOAligned(in_buffer);
OodleX_Shutdown(NULL,OodleX_Shutdown_LogLeaks_Yes,0);
return 0;
}
//===========================================================
**
example_lz_threadphased_threadfunc
Example of how to run a ThreadPhased decode yourself.
The basic idea of ThreadPhased decoding is that the OodleLZ_Decompress work on each BLOCK can be
split into two phases. This can be invoked by just calling OodleLZ_Decompress twice on the same
block, first with OodleLZ_Decode_ThreadPhase1, then with OodleLZ_Decode_ThreadPhase2.
To get parallelism, we can run the two phases on two separate threads.
The rule is that you must run the Phase2 on each block after the Phase1 for that block is done,
and with the same "decoderMem" pointer. The Phase2 decodes on all blocks must be done in
sequential order (unless they are Seek Resets).
The decoder memory used for OodleLZ_Decompress here must be larger than normal, of size
OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded().
Our thread model here is a two-thread circular buffer scan with semaphore signalling.
Thread 1 does :
For each block
Wait on sem_blocksavail to get an available circular buffer slot
Do Phase1 Decompress into a slot
Post sem_phase1done
Thread 2 does :
For each block
Wait on sem_phase1done to get a slot with phase1 decode done
Do Phase2 Decompress on that slot
Post sem_blocksavail to signal Thread 1 that this slot may be reused
**
struct example_lz_threadphased_threaddata
{
volatile OO_U32 * error_cancel;
OodleX_Semaphore * sem_consume;
OodleX_Semaphore * sem_produce;
OO_SINTa num_blocks;
OO_SINTa num_scratch_blocks;
OO_U8 * scratch_mem;
OO_SINTa scratch_block_size;
OO_U8 * rawBuf; OO_SINTa rawSize;
const OO_U8 * compBuf; OO_SINTa compSize;
OodleLZ_Decode_ThreadPhase threadPhase;
bool success;
};
#define THREAD_ERROR 0
#define THREAD_SUCCESS 1
// thread_function_DecodePhase
// can be used for both Phase1 and Phase2 !
static OO_U32 OODLE_CALLBACK example_lz_threadphased_threadfunc( void * user_data )
{
example_lz_threadphased_threaddata * data = (example_lz_threadphased_threaddata *)user_data;
const OO_U8 * compPtr = (OO_U8 *)(data->compBuf);
const OO_U8 * compEnd = compPtr + data->compSize;
OO_SINTa decoderMemSize = data->scratch_block_size;
OO_SINTa scratch_i = 0;
for(OO_SINTa block_pos = 0;block_pos < data->rawSize;block_pos += OODLELZ_BLOCK_LEN, scratch_i++)
{
// consume one :
OodleX_Semaphore_Wait(data->sem_consume);
// relaxed load of shared error_cancel variable ; Sem Wait acts as Acquire barrier
if ( *(data->error_cancel) ) return THREAD_ERROR;
if ( scratch_i == data->num_scratch_blocks ) scratch_i = 0;
OO_U8 * decoderMem = data->scratch_mem + scratch_i * decoderMemSize;
OO_U8 * chunk_ptr = (OO_U8 *)(data->rawBuf) + block_pos;
OO_SINTa block_len = OOEX_MIN((data->rawSize - block_pos),OODLELZ_BLOCK_LEN);
OO_BOOL indy;
OO_SINTa block_complen = OodleLZ_GetCompressedStepForRawStep(compPtr,compEnd-compPtr,block_pos,block_len,NULL,&indy);
if ( block_complen <= 0 )
{
// handle error
// relaxed store of shared variable; Sem Post acts as release barrier
*(data->error_cancel) = 1;
OodleX_Semaphore_Post(data->sem_produce);
return THREAD_ERROR;
}
OO_SINTa gotLen = OodleLZ_Decompress(compPtr,block_complen,chunk_ptr,block_len,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,
data->rawBuf,data->rawSize,
NULL,NULL,
decoderMem,decoderMemSize,
data->threadPhase
);
OOEX_ASSERT( gotLen == block_len+block_pos );
if ( gotLen != block_len+block_pos )
{
// handle error
// relaxed store of shared variable; Sem Post acts as release barrier
*(data->error_cancel) = 1;
OodleX_Semaphore_Post(data->sem_produce);
return THREAD_ERROR;
}
compPtr += block_complen;
OodleX_Semaphore_Post(data->sem_produce);
}
data->success = true;
return THREAD_SUCCESS;
}
static bool example_lz_threadphased_decompress(void * comp_buf,OO_SINTa comp_len,void * dec_buf,OO_SINTa dec_size,bool async)
{
*
circularBufferBlockCount is the number of circular buffer slots for the two threads to
communicate through
circularBufferBlockCount >= 2
higher is faster because it allows less synchronization of the two threads
lower means less memory required
circularBufferBlockCount >= 4 is reasonable
circularBufferBlockCount >= 6 is close to full speed
*
OO_SINTa circularBufferBlockCount = 6; // parameter
// async is an option for whether the whole operation is run async off this thread or not
// (eg. with 2 additional threads or 1 additional thread)
// check that the data contains a valid ThreadPhased compressor :
OodleLZ_Compressor compressor = OodleLZ_GetAllChunksCompressor(comp_buf,comp_len,dec_size);
if (! OodleLZ_Compressor_CanDecodeThreadPhased(compressor) )
{
OodleXLog_Printf_v1("Asked for ThreadPhase decode but ! OodleLZ_Compressor_CanDecodeThreadPhased\n");
return false;
}
// count the number of OODLELZ_BLOCK_LEN in the total size :
OO_SINTa nBlocks = (dec_size + OODLELZ_BLOCK_LEN-1)/OODLELZ_BLOCK_LEN;
// don't need more than nBlocks :
circularBufferBlockCount = OOEX_MIN(circularBufferBlockCount,nBlocks);
// allocate space for the scratch circular buffer :
OO_SINTa scratchBlockSize = OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded();
OO_SINTa scratchBufSize = scratchBlockSize * circularBufferBlockCount;
// NOTE in production you may wish to preallocate this memory
void * scratchBuf = OodleXMalloc(scratchBufSize);
//=========================================================
// set up the data needed for the thread phases
// NOTE if you want to make the whole decode asynchronous,
// you can't put this on the stack, you need to package it up in memory
// OodleX_Semaphore just initialize with 0 :
OodleX_Semaphore sem_blocksavail = 0;
OodleX_Semaphore sem_phase1done = 0;
OO_U32 shared_error_cancel = 0; // shared atomic variable
// starting state is that all circular buffer slots are available :
OodleX_Semaphore_Post(&sem_blocksavail,(OO_S32)circularBufferBlockCount);
example_lz_threadphased_threaddata td1 = { 0 };
td1.error_cancel = &shared_error_cancel;
td1.success = false;
td1.compBuf = (OO_U8 *)(comp_buf);
td1.compSize = comp_len;
td1.rawBuf = (OO_U8 *)(dec_buf);
td1.rawSize = dec_size;
td1.num_blocks = nBlocks;
td1.num_scratch_blocks = circularBufferBlockCount;
td1.scratch_mem = (OO_U8 *)(scratchBuf);
td1.scratch_block_size = scratchBlockSize;
// thread 1 waits for blocks to be available in the circular buffer
// and posts that phase1 is done
td1.threadPhase = OodleLZ_Decode_ThreadPhase1;
td1.sem_consume = &sem_blocksavail;
td1.sem_produce = &sem_phase1done;
// thread 2 waits for each block to reach phase1done
// and then posts that the block is reusable
// same as thread 1, just swap the semaphores :
example_lz_threadphased_threaddata td2;
td2 = td1;
td2.threadPhase = OodleLZ_Decode_ThreadPhase2;
td2.sem_consume = &sem_phase1done;
td2.sem_produce = &sem_blocksavail;
// create a thread to run Phase1 :
// NOTE : in production you probably don't want to create a thread every time you make
// this decompress call. Rather use an idle thread that's already created.
OodleX_Thread thread1 = OodleX_CreateThread(example_lz_threadphased_threadfunc,&td1);
// either run Phase2 asynchronously (on another thread) or synchronously on this thread :
if ( async )
{
OodleX_Thread thread2 = OodleX_CreateThread(example_lz_threadphased_threadfunc,&td2);
// ... current thread is now available while decompress runs on 2 other threads ...
// ... return and do other work ...
OodleX_WaitAndDestroyThread(thread2);
}
else
{
// synchronous version - just run Phase2 on this thread :
example_lz_threadphased_threadfunc(&td2);
}
OodleX_WaitAndDestroyThread(thread1);
// OodleX_Semaphore doesn't need cleanup
//===========================================================
OodleXFree(scratchBuf);
bool ok = td1.success && td2.success;
return ok;
}
| OodleXIOQ_WriteWholeFile_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_OpenWriteWholeFileClose_Async
- Defines
- Enumerants
- Structures
- Functions
- Typedefs
| OodleXIOQ_OpenForWriteTempName_Async |
|
|
Discussion
Start opening a file for writeParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Same as OodleXIOQ_OpenForWriteCreate_Async except that it creates a unique temp name to write to. The temp name starts with
nameBase, if given. Providing nameBase is helpful because it lets Oodle put the temp file in the same directory as the
final file name, which ensures that the final rename can be done without copying.
Should be used with OodleXIOQ_CloseFileRename_Async.
Writing to a temp name and then renaming over the desired output file only on successful completion is the
recommended way to write all files. It means you won't destroy the user's data by failing to successfully
overwrite a previously existing good file.
Discussion
Log some info about the platformDiscussion
This function should be called after OodleX_Init.
It prints some info to the Oodle Log about the Oodle build and your system.
This is a helpful thing to include in debug reports sent to RAD.
Discussion
Initialize OodleParameters
Return Value
Discussion
You must call OodleX_Init or OodleX_Init_NoThreads before any other Oodle function that you expect to work.
Pair with OodleX_Shutdown.
For minimal linkage, use OodleX_Init_NoThreads
enum OodleXFileMode
{
OodleXFileMode_Invalid = 0,
OodleXFileMode_Read = 1,
OodleXFileMode_WriteCreate = 2,
OodleXFileMode_Write = 2,
OodleXFileMode_ReadWrite = OodleXFileMode_Read|OodleXFileMode_Write,
OodleXFileMode_Force32 = 0x40000000
};
Discussion
FileMode used by OodleFile and such.
Not all OodleFile types support OodleXFileMode_ReadWrite
Enumerants
Discussion
Free a pointer allocated by OodleXMalloc or OodleXMallocAlignedParameters
| ptr | the pointer to free; allocated by OodleXMalloc or OodleXMallocAligned (must not be NULL)
|
| bytes | the size of the allocation as originally requested
|
Discussion
Providing the size of the malloc allows much faster freeing
Size must match the allocated size!
Uses the current OodleXMallocVTable ; this is an error if ptr was allocated from a different VTable.
| OodleXHandleCountdown_Alloc |
|
|
Discussion
Allocate an OodleXHandle to a simple data-less coutdown Parameters
Return Value
Discussion
initialCount should be greater than 0.
A Countdown is a simple handle which you can use to wait for completion of many tasks.
Use OodleXHandleCountdown_Decrement to decrement it. When it reaches 0 it becomes Done,
which means it satisfies an OodleX_Wait.
(A countdown is the same thing as a single-use Semaphore with an initial negative count)
If autoDelete is OodleXHandleAutoDelete_Yes , the Countdown handle
is deleted when count reaches zero. (a deleted handle also satisfies OodleX_Wait).
| Oodle Network compression |
|
|
| OodleXIOQ_ReadMallocWholeFile_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_ReadMallocWholeFile_Async
struct OodleConfigValues
{
OO_S32 m_OodleLZ_LW_LRM_step;
OO_S32 m_OodleLZ_LW_LRM_hashLength;
OO_S32 m_OodleLZ_LW_LRM_jumpbits;
OO_S32 m_OodleLZ_Decoder_Max_Stack_Size;
OO_S32 m_OodleLZ_Small_Buffer_LZ_Fallback_Size_Unused;
OO_S32 m_OodleLZ_BackwardsCompatible_MajorVersion;
OO_U32 m_oodle_header_version;
};
Discussion
OodleConfigValuesMembers
Discussion
Struct of user-settable low level config values. See Oodle_SetConfigValues.
May have different defaults per platform.
| OodleXMallocFailedHandler |
|
|
Discussion
OodleXMallocFailedHandler is called when a malloc fails
Return true to retry.
Discussion
free a pointer allocated by OodleXMalloc or OodleXMallocAligned Parameters
| ptr | the pointer to free (must not be NULL)
|
Discussion
Uses the current OodleXMallocVTable ; this is an error if ptr was allocated from a different VTable.
Prefer OodleXFreeSized whenever possible.
| FAQ: What are the Oodle deprecated compressors ? |
|
|
The new Oodle compressor family consists of Kraken, Leviathan, Mermaid, Selkie, and Hydra.
New Oodle users should only use the new sea monster family of compressors.
LZB16 is also supported but not recommended.
As of Oodle 2.9.0 the older compressors (hidden in OODLE_ALLOW_DEPRECATED_COMPRESSORS) are no
longer supported. They cannot be encoded or decoded.
If you have old data from Oodle 2.8.x or earlier, you can continue to use the old Oodle lib.
Discussion
Get the file name (OS name)Parameters
| file | the IOQFile to query
|
| pInto | filled with the file's OS name (UTF8)
|
| intoSize | number of bytes Oodle can write to pInto
|
Return Value
Discussion
Copies the OS name (UTF8) into pInto. This may not be the same as the name used when opening
the file, if that was a VFS name.
| example_lz_noallocs : Example demonstrating Oodle compression with no allocations |
|
|
Discussion
Oodle example_lz_noallocs
Very simple example of OodleLZ memory -> memory compression & decompression.
Shows how to use Oodle without any allocations done by Oodle. All memory needed is passed in by the client.
Uses stdio for file IO to load an input file.
See example_lz : Example demonstrating LZ compression and decompression for more advanced OodleLZ usage.
example_lz_noallocs only uses Oodle Core, no Oodle Ext
include oodle2.h and not oodle2x.h
If you want to use Oodle with no allocations, you cannot use OodleX. OodleX installs its own allocator into Oodle Core.
Do not use OodlePlugins_SetAllocators with OodleX.
See FAQ: How do I use Oodle with no allocator? and OodleCore_Plugins_SetAllocators
#include "../include/oodle2.h"
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz_noallocs
#endif
#include "read_whole_file.h"
void * OODLE_CALLBACK example_noallocs_MallocAligned_Logging(OO_SINTa size,OO_S32 alignment)
{
// malloc should not be called, log an error :
printf("ERROR : example_noallocs_MallocAligned_Logging called (size %d)\n",(int)size);
// use default Oodle MallocAligned as an easy way to get a cross-platform aligned malloc :
return OodleCore_Plugin_MallocAligned_Default(size,alignment);
}
void OODLE_CALLBACK example_noallocs_Free_Logging(void * ptr)
{
// malloc should not be called, log an error :
printf("ERROR : example_noallocs_Free_Logging called.\n");
OodleCore_Plugin_Free_Default(ptr);
}
extern "C" int main(int argc,char *argv[])
{
No initialization is needed for Oodle2 Core
we let Oodle Core use the default system plugins (the C stdlib)
To change them, use Core plugins
Install our own allocator plugins that log an error if called.
These should never be called. You could also disable them :
OodleCore_Plugins_SetAllocators(NULL,NULL);
but that is not recommended because it will cause a hard failure if Oodle ever needs memory.
OodleCore_Plugins_SetAllocators(example_noallocs_MallocAligned_Logging,example_noallocs_Free_Logging);
// get args :
const char * in_name;
if ( argc < 2 )
{
in_name = "r:\\testsets\\lztestset\\lzt02";
}
else
{
in_name = argv[1];
}
read input file using stdio
OO_SINTa length;
void * buf = read_whole_file(in_name,&length);
if ( ! buf )
{
fprintf(stderr,"couldn't open : %s\n",in_name);
return 10;
}
Run OodleLZ_Compress from memory (buf) to memory (compbuf)
Use the OodleLZ_Compressor_Kraken compressor. Kraken is an amazing balance of good compression and fast decode
speed. It should generally be your first choice.
Use OodleLZ_CompressionLevel_Normal level of effort in the encoder. Normal is a balance of encode speed and compression
ratio. Different levels trade off faster or slower encoding for compressed size. See OodleLZ_CompressionLevel.
See About OodleLZ for information on selection of the compression options.
This call is synchronous and not threaded; see example_lz : Example demonstrating LZ compression and decompression for an example using the async compression APIs.
OodleLZ_Compressor compressor = OodleLZ_Compressor_Kraken;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Normal;
//OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Optimal2; // optimals are OODLELZ_SCRATCH_MEM_NO_BOUND
// allocate memory big enough for compressed data :
void * compbuf = malloc( OodleLZ_GetCompressedBufferSizeNeeded(compressor,length) + sizeof(length) );
if ( compbuf == NULL )
return 10;
// allocate memory for encoder scratch :
OO_SINTa enc_scratch_size = OodleLZ_GetCompressScratchMemBound(compressor,level,length,NULL);
if ( enc_scratch_size == OODLELZ_SCRATCH_MEM_NO_BOUND )
{
// scratch cannot be bounded for this choice of compressor/level
// the allocator may be used!
// go ahead and give it 4 MB of scratch
enc_scratch_size = 4*1024*1024;
}
void * enc_scratch = malloc(enc_scratch_size);
if ( enc_scratch == NULL )
return 10;
char * compptr = (char *)compbuf;
memcpy(compptr,&length,sizeof(length));
compptr += sizeof(length);
// compress :
OO_SINTa complen = OodleLZ_Compress(compressor,buf,length,compptr,level,NULL,NULL,NULL,enc_scratch,enc_scratch_size);
compptr += complen;
// log about it :
// full compressed size also includes the header
printf("%s compressed %d -> %d (+%d)\n",in_name,(int)length,(int)complen,(int)sizeof(length));
// can free enc_scratch now
// enc_scratch can be reused for further compression
// but must be used by only one thread at a time
free(enc_scratch); enc_scratch = NULL;
Run OodleLZ_Decompress from memory (compbuf) to memory (decbuf)
Note that you must provide the exact decompressed size. OodleLZ data is headerless; store the size in
your own header.
OO_SINTa declength;
compptr = (char *)compbuf;
memcpy(&declength,compptr,sizeof(declength));
compptr += sizeof(declength);
assert( length == declength );
// malloc for decompressed buffer :
void * decbuf = malloc( declength );
if ( decbuf == NULL )
return 10;
OO_SINTa decoder_mem_size = OodleLZDecoder_MemorySizeNeeded(compressor,declength);
void * decoder_mem = malloc(decoder_mem_size);
if ( decoder_mem == NULL )
return 10;
// do the decompress :
OO_SINTa decompress_return = OodleLZ_Decompress(compptr,complen,decbuf,declength,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_Yes,OodleLZ_Verbosity_None,NULL,0,NULL,NULL,
decoder_mem,decoder_mem_size,
OodleLZ_Decode_Unthreaded);
// check it was successful :
assert( decompress_return == length );
assert( memcmp(buf,decbuf,length) == 0 );
if ( decompress_return != length )
return 10;
printf("decompessed successfully.\n");
And finish up. No shutdown is needed for Oodle2 Core.
// free all the memory :
free(buf);
free(compbuf);
free(decbuf);
free(decoder_mem);
return 0;
}
| t_fp_OodleNet_Plugin_Printf |
|
|
OODEFFUNC void( OODLE_CALLBACK t_fp_OodleNet_Plugin_Printf )( int verboseLevel,
const char * file,
int line,
const char * fmt,
. . . );Discussion
Function pointer to Oodle Core printfParameters
| verboseLevel | verbosity of the message; 0-2 ; lower = more important
|
| file | C file that sent the message
|
| line | C line that sent the message
|
| fmt | vararg printf format string
|
Discussion
The logging function installed here must parse varargs like printf.
verboseLevel may be used to omit verbose messages.
| OodleLZ_GetCompressedBufferSizeNeeded |
|
|
Discussion
Return the size you must malloc the compressed bufferParameters
| compressor | compressor used; OodleLZ_Compressor_Invalid to make it enough for any compressor
|
| rawSize | uncompressed size you will compress into this buffer
|
Discussion
The compBuf passed to OodleLZ_Compress must be allocated at least this big.
note this is actually larger than the maximum size of a compressed stream, it includes overrun padding.
Discussion
Get the Status of an async handleParameters
Return Value
Discussion
This function does not block. Returns OodleXStatus_Invalid if the handle was already deleted or does not exist.
Test status for done by checking >= OodleXStatus_Done, because that also includes Error.
Discussion
Shut down Oodle at app exit time.Parameters
| threadProfileLogName | (optional) if not NULL, and the ThreadProfiler is enabled, writes the threadprofiler output to this file name
|
| logLeaks | (optional) if true and the LeakTracker is enabled, logs any leaks or memory or handles
|
| allocStartCounter | (optional) initial counter for the LeakTrack log
|
| debugBreakOnLeaks | (optional) if there are any leaks, do a debug break
|
Discussion
Pair with OodleX_Init. No Oodle functions should be called after Shutdown.
Call Shutdown from the same thread that called Init.
Do not shutdown Oodle then init again. Only call Init and Shutdown once per run.
Discussion
Reset an OodleLZDecoder to restart at given posParameters
| decoder | the OodleLZDecoder, made by OodleLZDecoder_Create
|
| decPos | position to reset to; must be a multiple of OODLELZ_BLOCK_LEN
|
| decLen | (optional) if not zero, change the length of the data we expect to decode
|
Return Value
Discussion
If you are seeking in a packed stream, you must seek to a seek chunk reset point, as was made at compress time.
That is, OodleLZ_CompressOptions:seekChunkReset must have been true, and
decPos must be a multiple of OodleLZ_CompressOptions:seekChunkLen that was used at compress time.
You can use OodleLZ_GetChunkCompressor to verify that you are at a valid
independent chunk start point.
Discussion
Flush output to the logDiscussion
Close & reopen the log file to ensure data is on disk.
Useful when trying to debug a crash.
OodleXLog_Flush also prints queued messaged from threads, namely the IOQ Log
which you turn on with OodleXInitOptions:m_OodleInit_IOQ_Log. If you are using
the IOQ log, call OodleXLog_Flush once per frame or so when you want to see its
messages.
You can also use OODLEXLOG_AUTOFLUSH_THREADLOG in the log State, but that
only has affect when you do OodleLog Printfs.
| OodleXIOQ_OpenAndReadMallocWholeFileAndClose_Async |
|
|
Discussion
Start a high level IO request to open a file, allocate a buffer for a whole file and read itParameters
Return Value
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
Performs OodleXIOQ_OpenForRead_Async and OodleXIOQ_ReadMallocWholeFile_Async and OodleXIOQ_CloseFile_Async.
The OodleXHandle returned is to the RMWF operation; use OodleXIOQ_ReadMallocWholeFile_GetResult.
| OodleCore_Plugins_SetJobSystem |
|
|
Discussion
DEPRECATED use OodleCore_Plugins_SetJobSystemAndCount insteadDiscussion
See OodleCore_Plugins_SetJobSystemAndCount
Discussion
User-provided callback for threadsDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
| OodleLZ_GetFirstChunkCompressor |
|
|
Discussion
ask who compressed this chunkParameters
| compChunkPtr | pointer to compressed data; must be the start of compressed buffer, or a step of OODLELZ_BLOCK_LEN raw bytes
|
| compBufAvail | number of bytes at compChunkPtr available to read
|
| pIndependent | (optional) filled with a bool for whether this chunk is independent of predecessors
|
Return Value
Discussion
note this is only for this chunk - later chunks may have different compressors (eg. with Hydra)
if you compressed all chunks the same it's up to you to store that info in your header
Use OodleLZ_GetAllChunksCompressor for data that might be mixed compressors.
This replaces the deprecated function OodleLZ_GetChunkCompressor
returns OodleLZ_Compressor_Invalid if compBufAvail is too small or the chunk is corrupt
Discussion
Compute the number of seek chunksParameters
Return Value
| return | the number of seek chunks
|
Discussion
returns (rawLen+seekChunkLen-1)/seekChunkLen
struct OodleXFileOpsVTable
{
OodleXOSFile (OODLE_CALLBACK *sync_open)(const OodleXFileOpsVTable * vtable,const char * name,OodleXFileMode mode,OodleXFileOpenFlags flags, OO_S64 reserveSize, OO_S32 * pAlignmentRequired );
OO_BOOL (OODLE_CALLBACK *sync_close)(const OodleXFileOpsVTable * vtable,OodleXOSFile handle,const char * name, OO_S64 truncateSize );
OO_BOOL (OODLE_CALLBACK *sync_get_file_info)(const OodleXFileOpsVTable * vtable,OodleXOSFile handle,OodleXFileInfo * info);
OO_BOOL (OODLE_CALLBACK *sync_read)(const OodleXFileOpsVTable * vtable, OodleXOSFile handle, OO_S64 pos, OO_S32 size, void * memory, OO_S32 * pGotSize);
OO_BOOL (OODLE_CALLBACK *sync_write)(const OodleXFileOpsVTable * vtable,OodleXOSFile handle, OO_S64 pos, OO_S32 size, const void * memory);
OO_BOOL (OODLE_CALLBACK *sync_set_file_size)(const OodleXFileOpsVTable * vtable,OodleXOSFile handle, OO_S64 size);
OO_U32 (OODLE_CALLBACK *get_last_error)(const OodleXFileOpsVTable * vtable,OodleXOSFile handle);
OodleXError (OODLE_CALLBACK *get_error_enum)(const OodleXFileOpsVTable * vtable,OO_U32 err);
void (OODLE_CALLBACK *get_error_string)(const OodleXFileOpsVTable * vtable,OO_U32 err,char * pInto,int intoSize);
OO_BOOL (OODLE_CALLBACK *delete_file)(const OodleXFileOpsVTable * vtable,const char * name);
OO_BOOL (OODLE_CALLBACK *rename_file)(const OodleXFileOpsVTable * vtable,const char * fm,const char * to,OO_BOOL overwrite);
OodleXOSFileListing (OODLE_CALLBACK *listdir_open)(const OodleXFileOpsVTable * vtable,const char * dir);
void (OODLE_CALLBACK *listdir_close)(const OodleXFileOpsVTable * vtable,OodleXOSFileListing handle);
OO_BOOL (OODLE_CALLBACK *listdir_next)(const OodleXFileOpsVTable * vtable,OodleXOSFileListing handle,const char* path,char * name,int nameSize,OodleXFileInfo * info);
OO_BOOL (OODLE_CALLBACK *get_filename_info)(const OodleXFileOpsVTable * vtable,const char * name,OodleXFileInfo * info);
OO_BOOL (OODLE_CALLBACK *set_filename_info)(const OodleXFileOpsVTable * vtable,const char * name,OO_U32 flags,OO_U64 modtime);
OO_BOOL (OODLE_CALLBACK *force_writeable)(const OodleXFileOpsVTable * vtable,const char * name);
OO_BOOL (OODLE_CALLBACK *make_dir)(const OodleXFileOpsVTable * vtable,const char * name);
void * fileops_data;
};
Discussion
VTable struct of lowest level file opsMembers
| sync_open | open file in desired mode; reserveSize is for WriteCreate only |
| sync_close | close a file; must provide name if truncateSize is not -1 ; truncateSize only applies for files open for Write ; truncate may happen after closing, non-atomically |
| sync_get_file_info | fill out info |
| sync_read | read from a file at pos; pos, size & memory must all be aligned; fill out pGotSize; *pGotSize must == size if you return true unless you are at EOF |
| sync_write | write to a file at pos; pos, size & memory must all be aligned; must write full size if you return true |
| sync_set_file_size | set file size; size must be aligned! to set unaligned size, use truncateSize on close |
| get_last_error | returns an OS error code |
| get_error_enum | maps an OS error code to an OodleXError enum |
| get_error_string | takes OS error code, fills pInto (const OodleXFileOpsVTable * vtable,UTF8) |
| delete_file | delete by name |
| rename_file | rename by name ; overwrite existing only if passed true |
| listdir_open | returns 0 if dir does not exist |
| listdir_close | pair with listdir_open |
| listdir_next | listdir_next returns true if info is filled out ; name is UTF8 |
| get_filename_info | get info by name |
| set_filename_info | set OodleXFileInfo:flags and/or modtime (const OodleXFileOpsVTable * vtable,use OODLEX_FILEINFO_FLAG_INVALID and OODLEX_FILEINFO_MODTIME_INVALID to not change) |
| make_dir | make a dir |
| fileops_data | set by the creator the vtable and used however you like |
Discussion
IOQ acts through a VTable , which allows you to plug in your own functions to bottom level IO
| About Oodle Network Compression |
|
|
Oodle Network Compression provides realtime compression & decompression of game
packets.
The primary compressor which you probably want is OodleNetwork1. See
About OodleNetwork1 .
If you are evaluating Oodle Network, the main thing you need to do is to capture a large
data set for testing and training. See Capturing Training data for OodleNetwork
Note that network packet compression is separate from compression of static packages. eg. for sending
game updates or content, you would compress the data offline and just store it
compressed on your server. For that, you should use the normal Oodle LZ compressors.
See About OodleLZ.
| FAQ: OodleLZ_Decompress is failing how do I diagnose it? |
|
|
If you are getting OODLELZ_FAILED and having trouble isolating it, here are some suggestions.
For OodleLZ_Decompress to succeed, the compressed bytes must be correct, and the arguments you pass in
(compBufSize and rawLen) must be correct. The raw size that the data decompresses to is not stored in
the compressed data, it's up to you to pass that in, and if that's wrong it will show up as OODLELZ_FAILED.
The most common cause of this is that the compressed bytes are corrupted or weren't filled.
This can happen if the IO to fill them actually failed, or some mismatch about the number of compressed bytes
to load. It can happen if an async IO was used but it isn't actually done yet at the time of the Decompress
call (which would be a race).
There are a couple of things you can do to try to track it down :
Checksum the compressed data and verify it is valid before decompressing. You can either do your own checksums
or use the system in Oodle.
To use the system in Oodle, you must encode the data with OodleLZ_CompressOptions:sendQuantumCRCs enabled
(they are off by default), then in the OodleLZ_Decompress pass OodleLZ_CheckCRC_Yes.
Oodle's CRC's are on the compressed data and are checked before the actual decompression. This will still give
you an OODLELZ_FAILED but will log about crc failure.
To get the debug logs and perhaps more details :
Use the Debug build of Oodle. Verbose logs are not compiled into the release build.
Enable full log verbosity :
OodlePlugin_Printf_Default does not show verbose logs. Install OodlePlugin_Printf_Verbose with OodleCore_Plugins_SetPrintf.
(If using OodleX, doesn't do that, instead use OodleXLog_SetVerboseLevel.)
Another thing you can do :
When you get OODLELZ_FAILED from OodleLZ_Decompress, add a branch to your code that detects that case and writes out
the compressed buffer that was passed in, as well as all the arguments to the OodleLZ_Decompress function.
Also write out where you expected that compressed buffer to come from in a file.
Now go to the file that should contain that compressed data and diff it against the compressed data you logged out;
do they match? This can help to detect IO races and memory corruption.
Another thing that can be helpful to check for races :
After seeing an OODLELZ_FAILED return, pause and then repeat the same decompression again.
If it now succeeds, that could indicate a race.
OodleLZ is fully thread safe and does not access any globals, however the compressed buffer,
the output decompressed buffer, and the decoder memory passed in should only be used by the current
thread.
Note that races are not just possible in the compressed buffer that OodleLZ_Decompress reads;
the output uncompressed buffer is also read-write, so another thread writing in there could cause
an OODLELZ_FAILED, as could another thread using the passed in decoder scratch memory.
If you've done all these things and are still lost, contact me. I'll need you to send me the results of what you've
found in the above steps, as well as the log of all the function arguments & the compressed buffer contents passed to
OodleLZ_Decompress so I can reproduce the function call.
enum OodleLZ_Profile
{
OodleLZ_Profile_Main = 0,
OodleLZ_Profile_Reduced = 1,
OodleLZ_Profile_Force32 = 0x40000000
};
Discussion
Decode profile to target Enumerants
| OodleXIOQ_OpenForWriteCreate_Async |
|
|
Discussion
Start opening a file for writeParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
See many shared aspects of OodleXIOQ_OpenForRead_Async .
OpenForWrite opens files as create/truncate with exclusive access.
initialFileSize performs an initial pre-allocation of file space, same as OodleXIOQ_ReserveFileSizeForWrite_Async.
Pre-allocated file space has undefined (garbage) contents. Writes are faster to pre-allocated space.
WARNING : WriteCreate will overwrite (stomp) existing files by default. If you don't want that, pass
OodleXFileOpenFlags_WriteCreateDontStomp in fileOpenFlags.
enum OodleXLog_VerboseLevel
{
OodleXLog_Verbose_None = -1,
OodleXLog_Verbose_Minimal = 0,
OodleXLog_Verbose_Some = 1,
OodleXLog_Verbose_Lots = 2,
OodleXLog_Verbose_Force32 = 0x40000000
};
Discussion
Standard verbosity levels for use with OodleXLog_SetVerboseLevel
Enumerants
| OodleXLZ_ReadAndDecompress_Stream_Async |
|
|
OodleXHandle OodleXLZ_ReadAndDecompress_Stream_Async( OO_U32 asyncSelect,
const void * packedDataPtr,
OO_SINTa packedLen,
void * rawPtr,
OO_SINTa rawChunkLen,
OodleLZ_FuzzSafe fuzzSafe,
OodleLZ_CheckCRC checkCRC,
OodleLZ_Verbosity verbosity,
OodleDecompressCallback * pcb,
void * pcbData,
OodleXIOQFile readFile,
void * readBuf,
OO_S64 readStartPos,
OodleXHandle readPending,
OO_SINTa alreadyReadSize,
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start an async op to incrementally stream in data and decompressParameters
| asyncSelect | logical OR of OodleXAsyncSelect flags determine how the async is run (but Wide is ignored, this func is always narrow)
|
| packedDataPtr | pointer to start of compressed data
|
| packedLen | length of compressed data
|
| rawPtr | pointer to memory to decompress into
|
| rawChunkLen | lenght of raw data to decompress
|
| checkCRC | if OodleLZ_CheckCRC_Yes, the decompressor checks the crc to ensure data integrity
|
| verbosity | if not OodleLZ_Verbosity_None, will log some information
|
| pcb | OodleDecompressCallback called during decompression (NULL for none)
|
| pcbData | user data passed to pcb
|
| readFile | IOQ file to read compressed data from (0 for none)
|
| readBuf | pointer to memory where the reads from readFile should go (must be IO aligned)
|
| readStartPos | file position where readBuf starts (must be IO aligned)
|
| readPending | handle to previously fired read on the IOQ file
|
| alreadyReadSize | the number of bytes of readBuf that are already read (not the number in packedDataPtr)
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | handle to the operation
|
Discussion
OodleLZ_Async_Decompress_ReadStream :
coroutine streaming LZ decoder ;
does incremental file reads (optionally - readFile can be zero) ;
calls back decode progress so you can do incremental writes (or whatever) ;
does not need a seek table (reads raw LZ data).
OodleLZ_Async_Decompress_ReadStream is not "wide" (only one thread is used). It can be used to
overlap IO with decompression, but doesn't multi-thread decompression, even if the LZ data has seek chunks.
OodleXLZ_ReadAndDecompress_Stream_Async reads raw LZ data.
OodleXLZ_ReadAndDecompress_Stream_Async is mainly used when you want small granularity incremental callbacks;
if you only need OODLELZ_BLOCK_LEN callbacks, then OodleXLZ_Decompress_Narrow_Async is generally better,
and OodleXLZ_ReadAndDecompress_Wide_Async is fastest if you want "Wide" async decompression.
packedDataPtr should be somewhere inside readBuf (if the packed data is at the start of the file, they are equal).
That is, (packedDataPtr - readBuf + readStartPos) is the position in the file where compressed data starts.
Note that readBuf and readStartPos must be IO aligned, but packedDataPtr does not need to be, so to read compressed
data from a non-aligned
If provided, the OodleDecompressCallback is called as quanta of raw data are available. The callback
may be called more often than OODLELZ_BLOCK_LEN granularity.
Set OodleDecompressCallback to OodleDecompressCallback_WriteFile to perform a streaming read-compress-write.
Discussion
Find the seek entry that contains a raw positionParameters
Return Value
Discussion
returns the index of the chunk that contains rawPos
| OodleNetwork1_SelectDictionarySupported |
|
|
Discussion
Returns whether this version of the library can build new dictionaries.Discussion
This functionality is only available on host platforms, not embedded
devices, game consoles, phones etc.
Oodle2 Core LZ lossless data compression.
| OodleDecompressCallback_WriteFile_Data |
|
|
Discussion
A OodleDecompressCallback_WriteFile_Data for use with OodleDecompressCallback_WriteFileMembers
| file | the file handle to write to |
| lastWriteH | handle to the last write operation ; it's autodelete |
| closeH | handle to the file close operation ; NOT autoDelete |
| written | number of bytes written so far |
| doCloseFile | should the file be closed after the last write? |
Discussion
The OodleDecompressCallback_WriteFile_Data struct is passed as "userdata" to OodleDecompressCallback_WriteFile.
You must supply one as pcbData in functions that take a decompression callback.
Warning : if you make this object on the stack, ensure the lifetime is sufficient for the async operation!
| OodleLZ_CompressOptions_GetDefault |
|
|
Discussion
Provides a pointer to default compression optionsParameters
Discussion
Use to fill your own OodleLZ_CompressOptions then change individual fields.
| About OodleLZ ThreadPhased Decode |
|
|
About OodleLZ ThreadPhased Decode
The new Kraken compressor can be decoded with a new type of parallelism called "ThreadPhased".
(check OodleLZ_Compressor_CanDecodeThreadPhased)
ThreadPhased decoding works by running the decode operation per block in two phases. This allows 1-2X
speedup using two threads for decoding, typically around 33%-50% speedup. (Mermaid can decode ThreadPhased
too, but the benefit isn't as large as it is with Kraken)
ThreadPhased decoding can be done on the normal compressed data made with OodleLZ_Compress. You
don't need to prepare the data specially for ThreadPhased decoding, or break it into chunks with
the Oodle seek chunk reset system. (you can combine seek chunks and ThreadPhased decoding for
even more parallelism if you like).
Make compressed data by calling OodleLZ_Compress just like you normally would.
ThreadPhased decoding requires more memory than normal single-threaded decoding, because it needs
staging space for the two threads to communicate.
The easiest way to try ThreadPhased decoding is to use the helper in OodleX OodleXLZ_Decompress_ThreadPhased_Narrow_Async .
That runs a 2-thread decode on the OodleX Worker system, freeing the calling thread for other work.
example_lz_threadphased : Example of 2-thread ThreadPhased decoding includes a demonstration of that call.
The basic idea of ThreadPhased decoding is that the OodleLZ_Decompress work on each BLOCK can be
split into two phases. This can be invoked by just calling OodleLZ_Decompress twice on the same
block, first with OodleLZ_Decode_ThreadPhase1, then with OodleLZ_Decode_ThreadPhase2.
To get parallelism, we can run the two phases on two separate threads.
The rule is that you must run the Phase2 on each block after the Phase1 for that block is done,
and with the same "decoderMem" pointer. The Phase2 decodes on all blocks must be done in
sequential order (unless they are Seek Resets). The Phase1 decodes can be done in any order.
The decoder memory used for OodleLZ_Decompress here must be larger than normal, of size
OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded().
To perform a portion of a ThreadPhased decode, you simply call OodleLZ_Decompress , but with some
special argument values :
OodleLZ_Decompress(
compBuf,compBufferSize, // should be the compressed data for one block only
rawBuf,rawLen, // should be the destination uncompressed pointers for this block
checkCRc,verbosity,
decBufBase,decBufSize, // should be the whole destination buffer (or one whole seek chunk)
fpCallback,callbackUserData,
decoderMemory,decoderMemorySize, // must be provided, of size OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded()
threadPhase // set to OodleLZ_Decode_ThreadPhase1 or OodleLZ_Decode_ThreadPhase2
);
The decoderMemory must be allocated by the caller, and must be the same for each Phase on the block.
That is, don't use the decoderMemory for the next block in the same phase, instead use it for the
same block in the next phase. The decoderMemory is the place where the work of phase1 is passed
to phase2.
The two threads run through the blocks sequentially, passing the results of phase1 to the input of
phase2. Each phased block must get its own decoderMemory.
[phase1 block1] [phase1 block2] ...
| |
| |
[phase2 block1] [phase2 block2] ...
See example_lz_threadphased : Example of 2-thread ThreadPhased decoding for a full client-side implementation
| About Oodle on Windows UWP |
|
|
Oodle for Windows UWP contains the Core module only (no ext) and is provided as a DLL.
Oodle for Windows UWP is built with MSVC 2017 and the Windows 10 SDK.
lib/oo2core_winuwp32.lib
lib/oo2core_winuwp64.lib
redist/oo2core_5_winuwp32.dll
redist/oo2core_5_winuwp64.dll
The debug version of the DLL is also provided to help during development. It should not be redistributed.
C++ apps for UWP can use the import libs to access the DLL.
C# and Store apps may need to use the "LoadPackagedLibrary" mechanism.
What you do is add the DLL to the project as content (and make sure to check "deploy" in its properties).
Then you load it with "LoadPackagedLibrary" (not the normal LoadLibrary) to get an HMODULE , then you can do GetProcAddress to get func pointers.
If your UWP client has very limited Oodle usage, you could get away with only importing
the OodleLZ_Decompress function.
(extra step: if the UWP app is C#, not C++, then you must also tell it to deploy the VC runtime which it will not do by default)
| OodleXMalloc_SetFailedHandler |
|
|
Discussion
Install the OodleXMallocFailedHandler that will be usedParameters
| f | the function pointer to call (can be null for none)
|
Discussion
| OodleLZ_Compressor_UsesLargeWindow |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor is "LargeWindow" or not, meaning it can benefit from
a Long-Range-Matcher and windows larger than OODLELZ_BLOCK_LEN
enum OodleLZ_Jobify
{
OodleLZ_Jobify_Default = 0,
OodleLZ_Jobify_Disable = 1,
OodleLZ_Jobify_Normal = 2,
OodleLZ_Jobify_Aggressive = 3,
OodleLZ_Jobify_Count = 4,
OodleLZ_Jobify_Force32 = 0x40000000
};
Discussion
Controls the amount of internal threading in OodleLZ_Compress callsEnumerants
Discussion
Once you install a pluggable job system via OodleCore_Plugins_SetJobSystem, Oodle can internally break
heavy-weight compression tasks into smaller jobs that can run in parallel. This can speed up
compression of large blocks of data at Optimal1 and higher levels substantially.
The trade-off is that running more jobs concurrently rather than sequentially can greatly increase
memory requirements when there are multiple outstanding memory-intensive jobs.
OodleLZ_Jobify_Default lets the compressor decide; typically compressors will default to "Normal"
when a pluggable job system has been installed, and "Disable" otherwise.
OodleLZ_Jobify_Disable disables use of internal jobs entirely; all compression work is done on
the calling thread. This minimizes the amount of memory used, and is also appropriate when you're
getting parallelism in other ways, e.g. by running OodleLZ_Compress on many threads yourself.
OodleLZ_Jobify_Normal uses jobs to increase compressor parallelism and speeds up compression of
large blocks of data, but avoids handing out many concurrent jobs for tasks that are memory-intensive.
OodleLZ_Jobify_Aggressive will use concurrent jobs even for highly memory-intensive tasks. This
can speed up things further, but at a potentially significant increase in the amount of memory used
by Oodle.
| OodleXIOQ_SetFileSize_Async |
|
|
Discussion
Start a set-file-size requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Change the size of a file open for writing.
OodleXIOQ_SetFileSize_Async may align up size to the next sector boundary or OODLEX_IO_MAX_ALIGNMENT.
The contents of the file in the resized but unwritten area are undefined/garbage.
To write a file with non-aligned size, use OodleXIOQ_CloseFile_Async:truncateFileSize in OodleXIOQ_CloseFile_Async.
If the purpose of calling SetFileSize is to pre-reserve space to make writes go faster, then use OodleXIOQ_ReserveFileSizeForWrite_Async instead.
| OodleX_WaitDoneAllPending |
|
|
Discussion
Block on all pending operations being completedDiscussion
FlushAllAsync kills all parallelism and should generally only be used at shutdown or error handling.
FlushAllAsync is only guaranteed to stop pending handles that were fired before this call starts.
If new operations are created by other threads (or by existing pending operations) they may still be
pending when this call returns.
| FAQ: I write a file with IOQ but the contents are garbage? |
|
|
A very common error is using an async write, but failing to keep the buffer alive long enough.
For example, code like this :
OodleIOQFile file; // parameter
SINTa size; // parameter
void * buffer = OodleXMalloc_IOAligned(size);
// .. fill buffer ..
SINTa alignedSize = OodleAlignUp(size);
OodleIOQ_Write_Async(toFile,buffer,alignedSize,0,OodleXHandleAutoDelete_Yes);
OodleFree_IOAligned(buffer);
may free the buffer during the async IO. This may or may not produce an error. In some cases the write will
succeed but simply write junk data. (in some cases the write may succeed and write the buffer correctly; this is
a race condition and the results are unpredictable).
To fix this you must ensure the buffer is not freed until the write is done. Often you will want to OodleX_Wait on
the write being done for some other reason, so you can simply move the buffer free until after the Wait.
Another option is to enqueue the Free using OodleXIOQ_FreeBufferIOAligned_Async.
| OodleXUtil_ConvertUTF16ToUTF8 |
|
|
Discussion
Convert a UTF16 string to UTF8Parameters
| from | string to convert
|
| to | filled with the result
|
| toSize | number of characters availabe in to (not bytes!)
|
Discussion
Fill [ to , to + toSize ] with the UTF8 conversion of from (UTF16).
Oodle strings are all UTF8. This can be used to convert a string from UTF16 , for example
for use with the Windows wchar routines.
Note that OodleXUtil_ConvertUTF8ToUTF16 + OodleXUtil_ConvertUTF16ToUTF8 may not produce
the same output, because UTF encodings are not unique. Also note that Windows 8-bit names
(from "A" code page, though they are not ANSI or ASCII) are not UTF8.
Discussion
Provides a string naming a OodleLZ_Jobify enum
Discussion
Set the echo output fileParameters
| echo | the stdio FILE to output to
|
Discussion
You must also enable the flag OODLEXLOG_ECHO in OodleXLog_SetState if you want
output to the echo FILE.
You can call SetEcho(NULL) to disable echoing, or you can SetState and
disable the OODLEXLOG_ECHO flag.
Discussion
Get current working directoryParameters
Discussion
After the call, into ends in a path delim, so file names can be stuck on it with strcat.
Discussion
OS file handle abstract type
| OodleX_Init_ThreadProfilerInit |
|
|
Discussion
Get the func pointer for m_OodleInit_ThreadProfiler_funcptr
| OODLEX_FILE_CLOSE_NO_TRUNCATE_SIZE |
|
|
Discussion
Pass for truncateFileSize to OodleXIOQ_CloseFile_Async if you don't want it to truncate
| OODLEX_BUFFER_SIZE_DEFAULT |
|
|
Discussion
Pass to functions that want a buffer size to indicate the default should be used.Discussion
The buffer size used comes from OodleXConfigValues
| OodleXDecompressCallback_WriteFile_Data_Init |
|
|
Discussion
fills out an OodleDecompressCallback_WriteFile_Data structParameters
Discussion
Fills out the pcbData for use with OodleDecompressCallback_WriteFile.
Opens fileName for write with OodleXIOQ_OpenForWriteCreate_Async.
Discussion
Release OodleX TLS resources on the calling threadDiscussion
Call on a thread before it terminates to release resources that OodleX may have put in the TLS
of this thread.
The purpose of this is to avoid increasing memory use in code bases that create & destroy a lot of
threads for jobs. In that case, Oodle may allocate a bit of memory per thread and never free it,
which will add up over time.
In normal game code bases that create a fixed number or low number of threads, you should not
bother calling this.
NOTE : any use of OodleX functions on this thread after calling this may crash!
This should be the last thing called on this thread before it terminates or returns from
its thread function.
Discussion
Fire any requests which have not previously been startedDiscussion
If requests were enqueued with kick = false (don't start immediately), then they can be
started this way. Disabling auto-kick is good for performance when a very large number of
request are being created in a short period of time.
Discussion
convert the file name extension (max 4 chars) into a U32 for fast compares Parameters
| filename | file name with an extension after a period
|
Return Value
| return | a U32 with up to 4 chars of extension
|
Discussion
The return value is case-insensitive and does not include the dot.
| OodleNetwork1UDP_StateCompacted_MaxSize |
|
|
Discussion
Returns the size of memory required for an OodleNetwork1UDP_StateCompacted objectDiscussion
Shared and State are allocated with malloc( Size() )
| OodleLZ_CheckSeekTableCRCs |
|
|
Discussion
Check the CRC's in seekTable vs rawBufParameters
Return Value
| return | true if the CRC's check out
|
Discussion
Note that OodleLZ_Decompress option of OodleLZ_CheckCRC checks the CRC of compressed data,
this call checks the CRC of the raw (uncompressed) data.
OodleLZ data contains a CRC of the compressed data if it was made with OodleLZ_CompressOptions:sendQuantumCRCs.
The SeekTable contains a CRC of the raw data if it was made with OodleLZSeekTable_Flags_MakeRawCRCs.
Checking the CRC of compressed data is faster, but does not verify that the decompress succeeded.
| OodleNetwork1UDP_State_Compact_ForVersion |
|
|
Discussion
See OodleNetwork1UDP_State_Compact
*
* takes oodle_major_version to target
*
* Oodle Network Compacted state changed from major version 5 to 6 (eg 2.5.5 to 2.6.0)
*
| OodleLZ_GetCompressScratchMemBound |
|
|
Discussion
Return the maximum amount of scratch mem that will be needed by OodleLZ_CompressParameters
Discussion
If you pass scratch mem to OodleLZ_Compress of this size, it is gauranteed to do no allocations.
(normally if it runs out of scratch mem, it falls back to the installed allocator)
For rawLen pass at least the maximum size you will ever encode. If your data is divided into chunks,
pass the chunk size. If you will encode full buffers of unbounded size, pass -1.
The options must be the same as when you call OodleLZ_Compress
Some options and levels may not have simple finite bounds. Then OODLELZ_SCRATCH_MEM_NO_BOUND is returned
and the call to OodleLZ_Compress may use the allocator even if infinite scratch memory is provided.
Currently this applies to all the Optimal levels.
When OODLELZ_SCRATCH_MEM_NO_BOUND is returned, you can still pass in scratch mem which will be used before
going to the plugin allocator.
| OodleXIOQ_MakeAllDirs_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_MakeAllDirs_Async
| OodleNet_Plugins_SetJobSystem |
|
|
Discussion
DEPRECATED use OodleNet_Plugins_SetJobSystemAndCount insteadDiscussion
See OodleNet_Plugins_SetJobSystemAndCount
| Capturing Training data for OodleNetwork |
|
|
To evaluate Oodle Network fairly, you will need to capture network packets from a real play session.
Once you are using Oodle Network in production, you will need to make a large capture to train the
compressor that you ship.
If possible, capture from a real game session (perhaps from your QA department
playing the game), not a simulation using bots. Simulated captures can have patterns that don't
reflect real play.
The captured packets should be without any encryption or other compression
algorithms applied. Any already compressed data (such as zlib or jpeg or voice data)
should be excluded from the packet capture. Very large packets that you will compress with OodleLZ should also be excluded.
You should typically continue to use your heuristic bit-packing or delta scheme.
Try to capture at least 100 MB of packet data for the evalution. For final game training you should
capture more. Note that just capturing a very long single session is not generally helpful; you want to take a
broad sampling of many sessions over time to ensure that the capture is reflective of the whole spectrum of
packets that the game sends.
The better the training packets match the packets seen in the final game, the more compression there will be.
Mismatches are not a disaster, they simply mean that part of the dictionary is not useful for compression.
The example_packet : Example demonstrating network packet compression shipped with Oodle reads this file format :
packet.bin :
U32 [LE] : numbers of channels (num_channels)
repeatedly :
{
U32 [LE] : channel index in [0,num_channels-1]
U32 [LE] : number of bytes of data in this packet (num_bytes)
U8 * num_bytes : payload of this packet
}
though you may always change example_packet to read a different format.
For UDP, write num_channels = 1 and the channel index of all packets as 0.
The cleanest way to capture packets is to add code to your server to log them out immediately before sending.
If you can't change your server code, then something like tcpdump can be used, but you will have to strip
the protocol headers.
For your final shipping capture, you should try to capture packets from a wide variety of play sessions in
different levels, with different numbers of players, to get a broad sampling of what your network traffic
looks like. Then combine random portions of those captures to make the packet file that you use for Oodle
Network training. This ensures that you don't train on a non-representative set of data.
NOTE : the packets that you hold out for testing/training/dictionary should be a random selection of
packets, not linear chunks. You want each group of packets to be an independent random sampling of the
network traffic. Each group should span the range of different types of data you send. (example_packet : Example demonstrating network packet compression
includes one way of doing this)
NOTE : if you are compressing both upstream and downstream, those should generally have different dictionaries
and different trained states. Typically the nature of the network traffic up and down is very different, so
they should not be mixed together in a single capture. If you have very distinct network traffic phases
(such as, for example, a lobby or match-making phase and then a match play phase) then it may be advantageous
to separate those types of traffic for compression.
Contact oodle@radgametools.com with any questions
enum OodleXStatus
{
OodleXStatus_Invalid = 0,
OodleXStatus_Pending = 1,
OodleXStatus_Done = 2,
OodleXStatus_Error = 3,
OodleXStatus_Count = 4,
OodleXStatus_Force32 = 0x40000000
};
Discussion
OodleXStatus indicates the status of asynchronous weak reference handles.Enumerants
Discussion
The OodleXStatus generally increases in numeric value during its autoDelete.
Check status >= OodleXStatus_Done to test for completion (possibly error).
Not yet allocated : OodleXStatus_Invalid = 0
Fired off and still pending : OodleXStatus_Pending = 1
Completed (possibly in error) : OodleXStatus_Done = 2 or Error = 3
| OodleNetwork1UDP_StateCompacted |
|
|
Discussion
Opaque type for an OodleNetwork1UDP_StateCompactedDiscussion
Compacted version of OodleNetwork1UDP_State
Used to decrease the size of OodleNetwork1UDP_State for storage.
You cannot code with a OodleNetwork1UDP_StateCompacted.
Oodle2 Ext LZ lossless data compression.
Acknowledgements
The product names Oodle, Bink, Miles, Iggy, Granny and Telemetry are all copyrighted and trademarked by Epic Games Tools LLC. Printed and produced in the United States of America.
As a licensee of Oodle, you must abide by the terms set forth in your license agreement. Please refer to that agreement if you have any questions about what you may or may not do with this documentation or the software to which it pertains. Oodle is not copy protected, but it is copyrighted. We think our license agreements are fair, and that our software is reasonably priced for the quality and effort we have put into it. Using our software in ways other than allowed by your license agreement violates federal, civil, and criminal law. We rely primarily on your good faith not to violate our copyright; please respect it.
This software and documentation are provided "as is" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. In no event will Epic Games Tools LLC be liable to you for damages, including any general, special, incidental or consequential damages arising out of the use or inability to use the product (including, but not limited to, loss of data).
Oodle was written by Charles Bloom and Fabian Giesen.
The IDOC automatic documentation system used for Oodle was written by Sean Barrett.
The compression algorithms in Oodle benefitted from discussion with countless developers who have shared their thoughts and code with me over the years.
| OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN |
|
|
Discussion
compressed buffer must be sized to at least compLen+OODLENETWORK1_DECOMP_BUF_OVERREAD_LENDiscussion
(note that this is strictly less than OodleNetwork1_CompressedBufferSizeNeeded(rawLen))
| t_fp_OodleCore_Plugin_MallocAligned |
|
|
Discussion
Function pointer type for OodleMallocAlignedParameters
| bytes | number of bytes to allocate
|
| alignment | required alignment of returned pointer
|
Return Value
| return | pointer to memory allocated (must not be NULL)
|
Discussion
alignment will always be a power of two
alignment will always be >= OODLE_MALLOC_MINIMUM_ALIGNMENT
Discussion
Decode a packetParameters
| state | state of this compression channel; will be mutated
|
| shared | const shared compression context
|
| comp | compressed packet received
|
| compLen | size of compressed data
|
| raw | output decompressed packet
|
| rawLen | size of the packet to write
|
Return Value
Discussion
Decodes one packet. state is mutated, learning from this packet for future packets.
The rawLen provided here must match the length used in OodleNetwork1TCP_Encode when creating this compressed packet. The OodleNetwork1 data is headerless, it's up to you to send the packet decompressed size in your own header.
If corrupt data is detected, false is returned.
If the number of compressed bytes consumed does not match compLen, false is returned.
If the number of output bytes does not match rawLen, false is returned.
This function, however, does not do verify data integrity. It will return 'true' if the correct number of bytes are coded,
even if the data does not match.
The buffer comp must be allowed to read at least compLen + OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN bytes
(note that this is strictly less than OodleNetwork1_CompressedBufferSizeNeeded(rawLen) , so that may be used as well)
Discussion
OS file listing abstract type
| OODLELZ_LOCALDICTIONARYSIZE_MAX |
|
|
Discussion
Maximum value of maxLocalDictionarySize in OodleLZ_CompressOptions
| OodleXIOQ_OpenAndReadMallocWholeFile_Async |
|
|
Discussion
Start a high level IO request to open a file, allocate a buffer for a whole file and read itParameters
Return Value
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
Performs OodleXIOQ_OpenForRead_Async and OodleXIOQ_ReadMallocWholeFile_Async.
The OodleXHandle returned is to the RMWF operation; use OodleXIOQ_ReadMallocWholeFile_GetResult.
You will normally want to enqueue an OodleXIOQ_CloseFile_Async after this.
Discussion
Pairs with OodleLZDecoder_CreateDiscussion
You should always call Destroy even if you provided the memory for OodleLZDecoder_Create
enum OODLEX_FILEINFO_FLAGS
{
OODLEX_FILEINFO_FLAG_DIR = (1<<0),
OODLEX_FILEINFO_FLAG_READONLY = (1<<1),
OODLEX_FILEINFO_FLAG_HIDDEN = (1<<2),
OODLEX_FILEINFO_FLAG_SYMLINK = (1<<3),
OODLEX_FILEINFO_FLAG_TEMPORARY = (1<<4),
OODLEX_FILEINFO_FLAG_OFFLINE = (1<<5),
OODLEX_FILEINFO_FLAG_Force32 = 0x40000000
};
Discussion
Flags for OodleXFileInfo:flags
Enumerants
Discussion
Encode a packetParameters
Return Value
| return | length of output compressed data written to comp ; the returned compLen is strictly <= rawLen
|
Discussion
Encodes one packet. state is mutated, learning from this packet for future packets.
The returned compLen will never be greater than rawLen, because OodleNetwork1 won't send packets that expand under compression (it just sends them uncompressed) - however it may write further than that during the compression attempt. Do not use the returned compLen to check the size of the compressed buffer needed.
| OodleXIOQ_ReserveFileSizeForWrite_Async |
|
|
Discussion
Start a set-file-size request, if it helps write speed.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
If the purpose of calling SetFileSize is to pre-reserve space to make writes go faster, then use OodleXIOQ_ReserveFileSizeForWrite_Async instead.
ReserveFileSizeForWrite is the same as SetFileSize, but it uses some information about the platform and the file to decide whether the reserve
will help or not. This function might do nothing if it thinks that the writes will be faster with no reservation.
The contents of the file in the resized but unwritten area are undefined/garbage.
See OodleXIOQ_SetFileSize_Async for more.
| OODLENETWORK1_MAX_DICTIONARY_SIZE |
|
|
Discussion
Maximum size of dictionary for OodleNetwork1
| OodleXIOQ_OpenForRead_Async |
|
|
Discussion
Start opening a file for readParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
If vtable is NULL, the vtable specified by the VFS mapping is used.
The file name provided is automatically run through VFS-to-OS name mapping, if applicable.
Open returns a File ref right away for your convenience, but the file is not actually open for a little while.
You can however go ahead and queue more requests on the file reference before open is complete.
You cannot call things that require an open file, such as OodleXIOQ_GetInfo.
OpenForRead is always shared access.
To also perform an initial read, use OodleXIOQ_OpenAndRead_Async
| OodleLZ_Compressor_UsesWholeBlockQuantum |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor is "whole block quantum" ; must decode in steps of
OODLELZ_BLOCK_LEN , not OODLELZ_QUANTUM_LEN like others.
| OodleX_CorePlugin_WaitJob |
|
|
Discussion
Function to plug in the OodleX Worker system to OodleCore_Plugins_SetJobSystemDiscussion
NOTE : OodleX_Init does OodleCore_Plugins_SetJobSystem automatically.
Discussion
Allocate some memory with specified alignmentParameters
| bytes | the amount to allocate (must be > 0)
|
| alignment | the desired alignment
|
Return Value
| return | pointer to allocated memory
|
Discussion
alignment must be <= bytes.
alignment must be power of 2.
OodleXMalloc uses the installed OodleXMallocVTable.
Pointer will be aligned to at least OODLE_MALLOC_MINIMUM_ALIGNMENT.
If a malloc fails, any installed OodleXMallocFailedHandler will be called.
struct OodleLZ_SeekTable
{
OodleLZ_Compressor compressor;
OO_BOOL seekChunksIndependent;
OO_S64 totalRawLen;
OO_S64 totalCompLen;
OO_S32 numSeekChunks;
OO_S32 seekChunkLen;
OO_U32 * seekChunkCompLens;
OO_U32 * rawCRCs;
};
Discussion
A seek table, as created by OodleLZ_CreateSeekTableMembers
Discussion
An OodleLZ_SeekTable can be created from any OodleLZ compressed data.
It should be transmitted if you wish to do random-access or parallel decompression.
One way to transmit it is to use an "OOZ" file which stores the OodleLZ_SeekTable in its header.
Discussion
Allocate some memoryParameters
| bytes | the amount to allocate (must be > 0)
|
Return Value
| return | pointer to allocated memory
|
Discussion
OodleXMalloc uses the installed OodleXMallocVTable.
Pointer will be aligned to at least OODLE_MALLOC_MINIMUM_ALIGNMENT.
If a malloc fails, any installed OodleXMallocFailedHandler will be called.
Discussion
Function to plug in the OodleX Worker system to OodleCore_Plugins_SetJobSystemDiscussion
NOTE : OodleX_Init does OodleCore_Plugins_SetJobSystem automatically.
Discussion
Set OodleXConfigValuesParameters
| ptr | your desired OodleXConfigValues
|
Discussion
Sets the global OodleXConfigValues from your struct.
You should call OodleX_GetConfigValues to fill the struct, then change the values you
want to change, then call OodleX_SetConfigValues.
This should generally be done before doing anything with Oodle (eg. even before OodleX_Init).
Changing OodleXConfigValues after Oodle has started has undefined effects.
| OodleXIOQ_ForceWriteable_Async |
|
|
Discussion
Start a force-writeable file request.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Asynchronously make a file writeable/deletable. Useful if the file might have read-only or other-user permissions
and you want to modifity anyway.
A common use is to enqueue a OodleXIOQ_ForceWriteable_Async right before a DeleteFile or RenameFile.
| OodleXLog_GetVerboseLevel |
|
|
Discussion
Get the global verbose levelReturn Value
Discussion
See OodleXLog_SetVerboseLevel
| OodleXIOQ_GetInfoByName_Async |
|
|
Discussion
Start an asynchronous GetInfo requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Used to get an OodleXFileInfo without blocking the calling thread. Use OodleXIOQ_GetInfoByName_GetResult to
get the result.
| t_fp_OodleCore_Plugin_Free |
|
|
Discussion
Function pointer type for OodleFreeReturn Value
| return | pointer to memory to free
|
Discussion
| OodleXUtil_ConvertUTF8ToUTF16 |
|
|
Discussion
Convert a UTF8 string to UTF16Parameters
| from | string to convert
|
| to | filled with the result
|
| toSize | number of characters availabe in to (not bytes!)
|
Discussion
Fill [ to , to + toSize ] with the UTF16 conversion of from (UTF8).
Oodle strings are all UTF8. This can be used to convert a string to UTF16 , for example
for use with the Windows wchar routines.
Note that OodleXUtil_ConvertUTF8ToUTF16 + OodleXUtil_ConvertUTF16ToUTF8 may not produce
the same output, because UTF encodings are not unique. Also note that Windows 8-bit names
(from "A" code page, though they are not ANSI or ASCII) are not UTF8.
| OodleXHandleEvent_SetDone |
|
|
Discussion
Set an OodleXHandleEvent to OodleXStatus_DoneParameters
Discussion
The state transition from Pending->Done is one way. If the handle is OodleXHandleAutoDelete_Yes, it
goes away now.
enum OodleXAsyncSelect
{
OodleXAsyncSelect_None = 0,
OodleXAsyncSelect_Workers = 0x100,
OodleXAsyncSelect_NoFlagsMask = 0xFFF,
OodleXAsyncSelect_Wide = 0x1000,
OodleXAsyncSelect_Full = 0xFFFF,
OodleXAsyncSelect_All = OodleXAsyncSelect_Full,
OodleXAsyncSelect_Force32 = 0x40000000
};
Discussion
OodleXAsyncSelect are bit masks that can be combined to form an async selector.Enumerants
Discussion
The async selector tells an async operation like OodleXLZ_Decompress_Narrow_Async where it should run its decompress.
OodleXAsyncSelect_Wide means break the task into many smaller pieces that can be run simultaneously, and consume all
available runners to make the task complete as quickly as possible. If WIDE is not specified, then the default is "narrow",
that is run async but don't split the task for minimum latency. Mainly used with OodleXAsyncSelect_Workers ; WIDE means
create several smaller Worklet, while narrow creates just one Worklet that does the whole task.
OodleXAsyncSelect_Full provides the quickest completion of any one call, but perhaps more contention with other operations.
| example_lz_overlap : Example demonstrating parallel overlap with OodleLZ |
|
|
Discussion
Oodle example_lz_overlap
Demonstration of the benefit of overlapping IO with CPU work and parallelism in LZ decompression.
This example compresses a file, then repeatedly reads the compressed data and decompresses it, in several different ways.
There are two types of parallelism demonstrated here :
1. IO overlap. When reading and decompressing large files, you get minimum latency by
overlapping the IO with the decompress. This is done by reading the compressed data in smaller
chunks and decompressing each chunk (in parallel) as it is done.
2. Parallel "wide" decompression. OodleLZ with seek chunk resets can decompress using many
threads simultaneously.
Combining IO overlap and wide decompression is the fastest way to load compressed data.
#include "../include/oodle2x.h"
#include "ooex.h"
#ifdef _MSC_VER
#pragma warning(disable : 4127) // conditional is constant
#endif
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#include "make_example_input.h"
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz_overlap
#endif
extern "C" int main(int argc,char *argv[])
{
if ( ! OodleX_Init_Default(OODLE_HEADER_VERSION) )
//if ( ! OodleX_Init_Default(OODLE_HEADER_VERSION,OodleX_Init_GetDefaults_DebugSystems_Yes,OodleX_Init_GetDefaults_Threads_No) )
{
fprintf(stderr,"OodleX_Init failed.\n");
return 10;
}
Pass in the file name to compress. If you don't provide one, I'll make one.
Ideally give this example something big to work on, 20M - 100M , to get better charts.
// get args :
const char * inName;
if ( argc < 2 )
{
inName = "oodle_example_input_file";
make_example_input(inName);
}
else
{
inName = argv[1];
}
// we'll write to this file :
const char * compName = "oodle_example_overlap_comp";
OodleXLog_Printf_v1("compressing %s -> %s ...\n",inName,compName);
Set up the LZ compression options so that we make independent chunks
(seekChunkReset = true) and make them 1 MB each.
This hurts compression ratio (vs not chunking) but allows decompression of each chunk
to run in parallel.
Read the input file, compress it, and write it out :
In practice for maximum speed, you could read each chunk of the input and compress them
independently as each chunk async IO read finishes.
OO_SINTa inSize;
void * inBuf;
OO_SINTa compSize;
OodleLZ_SeekTable * seekTable;
OO_S64 inSize64;
inBuf = OodleXIOQ_ReadMallocWholeFile_AsyncAndWait(inName,&inSize64);
if ( inBuf == NULL )
{
OodleXLog_Printf_v0("ERROR couldn't load : %s\n",inName);
return 10;
}
inSize = OodleX_S64_to_SINTa_check(inSize64);
{ // scope for compBuf
void * compBuf = OodleXMalloc_IOAligned( OodleLZ_GetCompressedBufferSizeNeeded(lzCompressor,inSize) );
OO_U32 asyncSelect = OodleXAsyncSelect_All;
compSize = OodleXLZ_Compress_AsyncAndWait(asyncSelect,lzCompressor,inBuf,inSize,compBuf,lzLevel,&lzOptions);
if ( compSize <= 0 )
{
OodleXLog_Printf_v0("ERROR failed to compress\n");
return 10;
}
if ( ! OodleXIOQ_WriteWholeFile_AsyncAndWait(compName,compBuf,compSize) )
{
OodleXLog_Printf_v0("ERROR couldn't write : %s\n",compName);
return 10;
}
// log about it :
OodleXLog_Printf_v1("%s compressed %d -> %d\n",inName,(int)inSize,(int)compSize);
Make an OodleLZ_SeekTable on the compressed data for later use with parallel decompression.
You should store the seektable to disk with any header information, and load it before loading the
compressed bulk data. Oodle can do this for you with an "OOZ" file.
For this example we'll just keep the seekTable in memory. In real use you would write the
seek table with the compressed data.
Once we make the seekTable, we free compBuf - to get the compressed data again we will
have to read it from disk.
seekTable = OodleLZ_CreateSeekTable(OodleLZSeekTable_Flags_None,
lzOptions.seekChunkLen,NULL,inSize,compBuf,compSize);
OodleXFree_IOAligned(compBuf);
compBuf = NULL;
}
// we'll decompress into decompBuf
void * decompBuf = OodleXMalloc(inSize);
//===================================================================================
// prevent cheating :
memset(decompBuf,(int)rand(),inSize);
Read the files with unbuffered IO
We're mainly doing this here for consistent benchmarking so we can see IO times
otherwise the files would just come from the OS buffers and no show real IO time.
In practice you should usually use buffered IO. (OodleXFileOpenFlags_Default)
Now begins various ways to read the compressed file and decompress it.
Read-Decomp 1 :
Read the whole compressed file, synchronously.
When that's done, decompress the whole buffer, synchronously.
Simple, but stalls the main thread and gets no overlap of IO or parallelism in the decompress.
OodleXLog_Printf_v1("Doing read then decomp, synchronously on the main thread :\n");
{
void * compBuf = OodleXIOQ_ReadMallocWholeFile_AsyncAndWait(compName,NULL,fileOpenFlags);
if ( compBuf == NULL )
{
OodleXLog_Printf_v0("ReadMallocWholeFile failed on %s\n",compName);
return 10;
}
OodleLZ_Decompress(compBuf,compSize,decompBuf,inSize,OodleLZ_FuzzSafe_Yes);
// check it :
OOEX_ASSERT( memcmp(inBuf,decompBuf,inSize) == 0 );
OodleXFree_IOAligned(compBuf);
}
//===================================================================================
Read-Decomp 2 :
Read the whole compressed file, and fire off a full-buffer decompress, scheduled
to run automatically when the read is done.
This has the same latency as method 1, but is a single async operation so the main
thread can do something else the whole time.
OodleXLog_Printf_v1("Doing read then decomp, through a job chain :\n");
// prevent cheating :
memset(decompBuf,(int)rand(),inSize);
{
// for simplicity, we'll just malloc compBuf with our known compSize
// more generally if you didn't know compSize, you would have to use an OodleWork coroutine
// to first open the file, get the size, do the malloc, do the read, then the decompress
void * compBuf = OodleXMalloc_IOAligned(compSize);
// make an IO request to open and read the whole file :
OodleXIOQFile compFile;
OodleXHandle openAndReadH = OodleXIOQ_OpenAndRead_Async(&compFile,compName,compBuf,OodleX_IOAlignUpSINTa(compSize),0,fileOpenFlags,0,OodleXHandleAutoDelete_Yes);
// go ahead and enqueue a Close to follow the OpenAndRead :
OodleXIOQ_CloseFile_Async(compFile,OODLEX_FILE_CLOSE_NO_TRUNCATE_SIZE,OodleXHandleAutoDelete_Yes);
// the decompress depends on the OpenAndRead - it will run only when that is done
// dependencies are an array, but we have only one, so just point at it :
const OodleXHandle * deps = &openAndReadH;
int num_deps = 1;
OodleXHandle decompH = OodleXLZ_Decompress_Narrow_Async(OodleXAsyncSelect_Full,compBuf,compSize,decompBuf,inSize,
OodleLZ_FuzzSafe_No,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,
0,0,0,0,0,0,OodleLZ_Decode_Unthreaded,0,0,0,0,
OodleXHandleAutoDelete_No,
deps,num_deps);
// ... main thread can do other work here ...
OodleXStatus st = OodleX_WaitAndDelete(decompH);
if ( st != OodleXStatus_Done )
{
OodleXLog_Printf_v0("OodleXLZ_Decompress_Narrow_Async failed!\n");
return 10;
}
// check it :
OOEX_ASSERT( memcmp(inBuf,decompBuf,inSize) == 0 );
OodleXFree_IOAligned(compBuf);
}
//===================================================================================
Read-Decomp 3 :
Read and decompress with IO overlap, but only using a single thread (not "wide").
OodleXLZ_ReadAndDecompress_Stream_Async is an API provided to do IO overlap with decompression.
It's something you could easily write yourself in Oodle. It uses a coroutine to do IO on chunks
and then decompress the chunks as they arive. It tries to always be doing the IO for the next chunk
while decompressing the current chunk.
OodleXLog_Printf_v1("Doing read and decomp simultaneously :\n");
// prevent cheating :
memset(decompBuf,(int)rand(),inSize);
{
void * compBuf = OodleXMalloc_IOAligned(compSize);
Enqueue an open request + read an initial small chunk into compBuf
We don't wait on the Open, but immediately start the OodleXLZ_ReadAndDecompress_Stream_Async operation,
and pass it the openAndRead handle.
OodleXIOQFile compFile;
OO_SINTa initialReadSize = OodleX_IOAlignUpSINTa( OOEX_MIN(compSize,512*1024) );
OodleXHandle openAndReadH = OodleXIOQ_OpenAndRead_Async(&compFile,compName,compBuf,initialReadSize,0,fileOpenFlags);
OodleXHandle readAndDecomp = OodleXLZ_ReadAndDecompress_Stream_Async(OodleXAsyncSelect_Full,
compBuf,compSize,decompBuf,inSize,
OodleLZ_FuzzSafe_No,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,0,0,
compFile,compBuf,0,openAndReadH,initialReadSize);
// enqueue a CloseFile with a dependency on the ReadAndDecomp operation :
OodleXIOQ_CloseFile_Async(compFile,OODLEX_FILE_CLOSE_NO_TRUNCATE_SIZE,OodleXHandleAutoDelete_Yes,OodleXPriority_Normal,&readAndDecomp,1);
// ... main thread can do other work ...
OodleXStatus st = OodleX_WaitAndDelete(readAndDecomp);
if ( st != OodleXStatus_Done )
{
OodleXLog_Printf_v0("ReadAndDecomp failed\n");
return 10;
}
// check it :
OOEX_ASSERT( memcmp(inBuf,decompBuf,inSize) == 0 );
OodleXFree_IOAligned(compBuf);
}
//===================================================================================
Read-Decomp 4 :
Read and decompress with IO overlap, and using all worker threads ("wide").
ReadAndDecompress_Wide needs the seekTable to find the compressed block boundaries. Normally you
would have to send that in a file (see the OOZ APIs if you want Oodle to do it for you). Here
we just use the seekTable we made earlier and kept in memory.
OodleXLog_Printf_v1("Doing read and decomp wide :\n");
// prevent cheating :
memset(decompBuf,(int)rand(),inSize);
{
void * compBuf = OodleXMalloc_IOAligned(compSize);
// open compressed file and do an initial read :
OodleXIOQFile compFile;
OO_SINTa initialReadSize = OodleX_IOAlignUpSINTa( OOEX_MIN(compSize,256*1024) );
OodleXHandle openAndReadH = OodleXIOQ_OpenAndRead_Async(&compFile,compName,compBuf,initialReadSize,0,fileOpenFlags);
// instead of waiting on openAndReadH here, you could pass it as a dependency
// to the OodleXLZ_ReadAndDecompress_Wide_Async function to make the whole sequence async
// but we'll just stall here for simplicity in this example :
OodleXStatus st = OodleX_WaitAndDelete(openAndReadH);
if ( st != OodleXStatus_Done )
{
OodleXLog_Printf_v0("OpenAndRead failed\n");
return 10;
}
// fire off the read-and-decomp job :
OodleXHandle readAndDecomp = OodleXLZ_ReadAndDecompress_Wide_Async(OodleXAsyncSelect_Full,
seekTable,compBuf,compSize,initialReadSize,compFile,0,
decompBuf,inSize,
OodleLZ_FuzzSafe_No,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None);
// enqueue a CloseFile with a dependency on the ReadAndDecomp operation :
OodleXIOQ_CloseFile_Async(compFile,OODLEX_FILE_CLOSE_NO_TRUNCATE_SIZE,OodleXHandleAutoDelete_Yes,OodleXPriority_Normal,&readAndDecomp,1);
// ... main thread can do other work ...
st = OodleX_WaitAndDelete(readAndDecomp);
if ( st != OodleXStatus_Done )
{
OodleXLog_Printf_v0("ReadAndDecomp failed\n");
return 10;
}
// check it :
OOEX_ASSERT( memcmp(inBuf,decompBuf,inSize) == 0 );
OodleXFree_IOAligned(compBuf);
}
//===================================================================================
// all done, clean up :
OodleLZ_FreeSeekTable(seekTable);
OodleXFree(decompBuf);
OodleXFree_IOAligned(inBuf);
OodleX_Shutdown();
return 0;
}
Discussion
Get OodleXConfigValuesParameters
| ptr | filled with OodleXConfigValues
|
Discussion
Gets the current OodleXConfigValues.
May be different per platform.
| OodleXMalloc_GetVTable_OS |
|
|
Discussion
get an OodleXMalloc VTable that contains allocators based on the OS system allocators Parameters
Discussion
This is usually used to set OodleXInitOptions:m_pBaseVTable
| OodleDecompressCallback_WriteFile |
|
|
Discussion
A OodleDecompressCallback which writes the decompressed data to a fileParameters
Discussion
OodleDecompressCallback is called incrementally during decompression.
This is provided as a convenience for use as an OodleDecompressCallback in functions that take that callback,
such as OodleXLZ_ReadAndDecompress_Stream_Async.
NOTE : you typically need to do OodleX_WaitAndDelete on the closeH from OodleDecompressCallback_WriteFile_Data
Discussion
Compute a valid seekChunkLenParameters
Return Value
Discussion
Returns a seekChunkLen which is close to (rawLen/desiredSeekPointCount) but is a power of two multiple of OODLELZ_BLOCK_LEN
desiredSeekPointCount = 16 is good for parallel decompression.
(OODLELZ_SEEKPOINTCOUNT_DEFAULT)
| OodleNetwork1_Shared_Size |
|
|
Discussion
Returns the size of memory required for an OodleNetwork1_Shared objectParameters
Discussion
Shared and State are allocated with malloc( Size() )
| Third Party License Notices |
|
|
Oodle uses "ConvertUTF" by Unicode, Inc. License notice follows :
* Copyright 2001-2004 Unicode, Inc.
*
* Disclaimer
*
* This source code is provided as is by Unicode, Inc. No claims are
* made as to fitness for any particular purpose. No warranties of any
* kind are expressed or implied. The recipient agrees to determine
* applicability of information provided. If this file has been
* purchased on magnetic or optical media from Unicode, Inc., the
* sole remedy for any claim will be exchange of defective media
* within 90 days of receipt.
*
* Limitations on Rights to Redistribute This Code
*
* Unicode, Inc. hereby grants the right to freely use the information
* supplied in this file in the creation of products supporting the
* Unicode Standard, and to make copies of this file in any form
* for internal or external distribution as long as this notice
* remains attached.
The Oodle examples are shipped with some code from cbloom.com ; the code is Public Domain.
OodleXHandle is the generic handle to an async operation. Every async operation
performed by Oodle corresponds to an OodleXHandle.
(See also About Oodle Ext)
OodleXHandle is a user-mode waitable handle. It is very light weight; it provides
lock-free status and liveness checks. It provides real waits without polling or "thread thrashing"
or timed Sleeps, all of which hurt thread performance greatly.
An OodleXHandle is a "weak reference". That means that the object it refers to can
be deleted at any time, and the handle stays a valid unique reference to that object.
If you try to do an operation (like OodleX_GetStatus) on an object that was deleted, it will
return OodleXStatus_Invalid.
In general, Oodle considers a handle that doesn't exist to be equivalent to a handle that's Done.
eg. if you start an operation with a dependency on a handle that doesn't exist, the operation can
start immediately.
All operations on OodleXHandle are thread safe and atomic. eg. if two threads try to
delete a handle at the same time, only one of them will get the delete, and the other will
either see the handle before the deletion or after the deletion (not during).
You can always use OodleX_Wait or OodleX_GetStatus on any handle. You can combine handles from
various subsystems and OodleX_WaitAll on all of them. There are also subsystem-specific functions,
and it's up to you to use them correctly. For example if you call OodleAsyncGroup_Lock on a
handle that is not an OodleAsyncGroup, it will fail in an undefined way.
OodleXHandles always have a strict progression through their autoDelete. When they
are first created they are OodleXStatus_Pending. They stay Pending while they are
being run, and then they are set to OodleXStatus_Done or OodleXStatus_Error.
(to check completion you should not use == OodleXStatus_Done, instead use !=
OodleXStatus_Pending). A call to OodleX_Wait on a handle will return when status is
not Pending. Then a handle is deleted, and any further checks on it return OodleXStatus_Invalid.
A single handle can never go from Done back to Pending - if there is more work to do,
you must create a new handle for the next bit of work.
OodleXHandles with a autoDelete of OodleXHandleAutoDelete_Yes may never actually
be set to OodleXStatus_Done, they go directly from OodleXStatus_Pending to deleted.
The state transition from pending to deleted still triggers any waits on that handle.
Oodle does not delete handles for you (unless you tell it to). OodleXHandle lifetimes are managed by the client.
When you fire off an async operation,
you often get the option of whether the handle should be "autoDelete" or not. If a handle
is autoDelete, as soon as its status becomes not-Pending (Done or Error), the handle is deleted by the system.
A deleted handle is "not Pending" so it satisfies OodleX_Wait, but you can't check if it failed
and you can't get any results from the operation. AutoDelete operations are "fire and forget";
they are convenient because you don't have to free the handle, but can be hard to use safely.
The other option for handle lifetimes is OodleXHandleAutoDelete_No (the default) ; if a handle is not AutoDelete,
then at some point you must free it. This allows you to get results and check for errors before
it is deleted. The most common way to delete a handle is via OodleX_Wait with OodleXHandleDeleteIfDone_Yes.
The other standard way is via handle-specific completion functions, such as Oodle_CopyFile_Wait_GetResult.
Deleting handles while they are still pending is undefined in general; it is possible in some
specific situations. (it can be done by calling a Cancel function)
If handles are not deleted, they will eventually take all the slots in the handle table.
Handle leaks can be found with OodleLeakTrack .
All OodleXHandles must be able to fit in a static table that is created by OodleInit.
The size is determined by OodleXInitOptions:m_num_handles_log2 (times 2048 since Oodle 2.9.7). If you run out of handles
during execution, Oodle will fail catastrophically. See FAQ: I ran out of OodleHandle table slots; what do I do?.
This section applies to Win32 Platforms. See also About Oodle on Windows UWP for UWP / WinRT / Windows App / Phone platforms.
Oodle on Windows is now built with MSVC 2017. It is compatible with MSVC 2015+.
Oodle on Windows is shipped as a DLL. This allows it to avoid linker conflicts with your application,
primarily caused by choice of C runtime. Each major rev of Oodle has a new DLL name, so there is no possibility
of importing a DLL version that doesn't match what you app was built with.
To ensure the DLL version matches your app, OodleX_Init or Oodle_CheckVersion check the version number and will fail if they are
incompatible.
To build with the Oodle DLL on Windows you link your app with just one of these libs :
lib/oo2core_win32.lib
lib/oo2core_win64.lib
lib/oo2ext_win32.lib
lib/oo2ext_win64.lib
and ship your app with one of these dll's :
redist/oo2core_##_win32.dll
redist/oo2ext_##_win32.dll
redist/oo2core_##_win64.dll
redist/oo2ext_##_win64.dll
Where ## is the major version number of Oodle. The libs are import libs that you link with your game; they know the name of the appropriate
DLL to load. The DLLs in the redist directory may be shipped with your game.
The debug build of the Oodle DLL is also provided. Generally the release build of Oodle should be linked with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems. Debug versions of the DLLs are
provided in the "redistdebug" directory. These should not be shipped with your game.
static libs on Windows are now provided as an option. These may be incompatible with your build due to
choice of compiler flags. If the static libs are incompatible, please use the DLL option to resolve the
problem. Unfortunately with static libs it is impossible to ensure broad link time compatibility.
To use the static libs, link with a lib from the "static_lib/" directory.
NOTE! WARNING! IMPORTANT! For users of OodleX DLL :
On Windows, the oo2ext DLL contains both OodleX and Oodle Core. Don't
link with both oo2core and oo2ext, just one or the other.
(on all other platforms, oo2ext is an addition to oo2core and you need both if you use
OodleX)
DO NOT link both core and ext on Windows - if you like Ext it includes a copy of Core.
Additionally linking Core is NOT benign, it will cause Ext to fail to set up the Core plugins consistently.
To build an Oodle Example :
- use MSVC 2015+
- make a new console project for x64, make it empty
- add an example cpp file
- add the oodle release 64 bit lib , either from lib/ or static_lib/
- if you use the DLL import lib, change your run working directory to the redist dir
Oodle assumes that Windows machines have SSE2. Newer processor features are detected dynamically.
The OodleX log file is written to c:\oodlelogs on Windows unless you have set m_OodleInit_Log_FileName.
NOTE : if you are using Core only on a Microsoft target, note that the default core log function only goes to
OutputDebugString - not stdio - so you may not see any error messages unless you are running in the debugger.
Either use Oodle X or install your own logger to get messages however you want.
| FAQ: How much memory do the Oodle compressors use ? |
|
|
The Oodle decoders need an input buffer and an output buffer which you provide.
In addition to that, they need an OodleLZDecoder object which is a fixed size. The size is
reported by OodleLZDecoder_MemorySizeNeeded so you can allocate the memory yourself if you like
(or put it on the stack), and Oodle decoding will then do no allocations. When you call
OodleLZ_Decompress without provided memory, this allocation is done for you.
(The input & output buffer can be circular to reduce memory use if you are streaming a file
through the decoder without ever holding the whole compressed or decompressed file in memory,
but that's not recommended for normal use as it reduces performance)
The amount of memory needed to decode each OodleLZ_Compressor is different.
For Kraken, Mermaid, Selkie & Leviathan the current (2.7) OodleLZDecoder_MemorySizeNeeded is
442552 bytes (but don't hard code that value, it could change!)
This Decoder object scratch is used during the decode call, but not retained. So you can reuse
the same memory for an unrelated (but not simultaneous on another thread) decode.
The decoder object memory can be passed in to OodleLZ_Decompress using the decoderMemory
and decoderMemorySize arguments. If you pass in this memory, OodleLZ_Decompress will do no
allocations.
The memory use needed for encoding is a different story.
The memory use for encoding generally goes up at higher compression levels. It can be quite
small at levels Fast and VeryFast, and can be quite high at levels Optimal2 and higher.
It's always recommended to run the Optimal encoders in 64-bit on machines with plenty of RAM
(at least 4 GB and ideally 8 GB).
The encoders now accept passed-in memory which they will use before trying to allocate. If they
run out of the scratch you passed in, they will call the installed Malloc and expect to get memory
back from it. They do not handle Malloc failure gracefully. You should always provide a plugin
allocator even if you expect it not to be used.
See FAQ: How do I limit the encoder memory use?
| OODLEX_ASYNC_HANDLE_PENDING |
|
|
Discussion
OodleXHandle to a special always-pending handle. This is for Oodle internal use only.
Calls to OodleX_GetStatus on this handle value will return &OodleXStatus_Pending.
This is designed for use with OodleAsyncGroup. See OodleAsyncGroup_ChangePending.
Calling OodleX_Wait on this handle is a deadlock.
This handle must not be deleted! Do not call OodleX_Wait on it with deleteIfDone = true.
OodleXMalloc is the allocator used internally by OodleX to allocate memory.
NOTE : if you use Oodle Ext , OodleXMalloc must be initialized for OodleX internal use. If you use Oodle Core, you do not need to use OodleXMalloc, you may plug in your own malloc. You cannot use your own malloc for Oodle Core if you are using OodleX - they cannot use different allocators. See Core plugins if you only need to change the Oodle Core malloc.
You may wish to point OodleX at your own allocator. This is done by installing an OodleXMallocVTable in OodleX_Init.
The OodleX allocator provides an implementation for the malloc needed by Oodle Core. It also adds functionality for doing IO-aligned allocations (required for async IO on some platforms).
Most of the low-level Oodle Core runtime calls (like OodleLZDecoder_Create) can be optionally passed in
pre-allocated memory so that they don't do any allocations at all.
Sometimes OodleX does an allocation for you and passes back the pointer, in which case
you must free it with OodleXFree.
There are two types of memory used in OodleX :
The low level async IO systems (primarily OodleXIOQ ; see About OodleIOQ ) require IO-aligned memory. You do not necessarily
have to use OodleXMalloc_IOAligned to get this memory, you can use your own malloc, but then you must ensure OODLEX_IO_MAX_ALIGNMENT.
If you ask OodleX to do the allocation for you (eg. by calling something like OodleXIOQ_OpenAndReadMallocWholeFile_Async), then
you must free the pointer with OodleXFree_IOAligned.
If you don't provide an allocator, OodleX will install its own, which calls through to the low level system allocator.
OodleX expects malloc to never fail. It doesn't necessarily return clean errors if it runs out of memory deep in a function.
If the installed OodleXMalloc returns NULL, then OodleX calls the OodleXMallocFailedHandler that's set by OodleXMalloc_SetFailedHandler.
In the FailedHandler you could abort your game, or try to free up memory, and then return true, in which case OodleX will try the
allocation again.
For clients interested in replacing OodleXMalloc with their own implementation, read on.
All OodleX allocations go through an OodleXMallocVTable. To install your own allocator, you must implement the functions needed and
fill out an OodleXMallocVTable with pointers to your functions. Then set OodleXInitOptions:m_pBaseVTable to your own VTable
before calling OodleX_Init. Generally you don't want to change the installed Malloc VTable between the calls to OodleX_Init and
OodleX_Shutdown - you should install one in Init and not change it.
By default, OodleXInitOptions has m_OodleInit_LockFreePageAllocator set to true; that puts the lock free page allocator on top of your
allocator. If you don't want that, then set it to false.
OodleXMalloc works by calling OodleXMallocCall on the global installed VTable. OodleXMallocCall can be used to invoke a malloc from any
vtable.
See OodleXMallocVTable for notes on how each function you provide must behave.
The functions you provide must all be thread safe.
One note on m_pMallocBig : you have choice about how to implement this. You can just do a malloc that provides memory aligned
to OODLEX_IO_MAX_ALIGNMENT. If you do that, then OodleXMalloc_IOAligned will simply call directly to your m_pMallocBig.
Alternatively, m_pMallocBig can be the system large page alloc (eg. on Windows it would be VirtualAlloc).
If you do that, then OodleXMalloc_IOAligned will allocate from portions of the "Big" pages.
The default OodleXMalloc sets m_pBaseVTable to the OS allocators (not clib's malloc/free, but the lower level allocators; for
example on Windows it uses HeapAlloc for m_pMalloc and VirtualAlloc for m_pMallocBig). The default then also has m_OodleInit_LockFreePageAllocator
set to true, which creates a layered OodleXMallocVTable which uses its own lock-free page allocator for small allocs, and passes through
large allocs to the base VTable (the OS allocators).
struct OodleXFileInfo
{
OO_U32 flags;
OO_U32 pad;
OO_S64 size;
OO_U64 modTime;
};
Members
| OODLEX_FILEINFO_FLAG_INVALID |
|
|
Discussion
Invalid value for OodleXFileInfo:flags
| OodleNetwork1UDP_State_Uncompact_ForVersion |
|
|
Discussion
See OodleNetwork1UDP_State_Uncompact
*
* takes oodle_major_version to target
*
* Oodle Network Compacted state changed from major version 5 to 6 (eg 2.5.5 to 2.6.0)
*
| OODLELZ_BLOCK_MAX_COMPLEN |
|
|
Discussion
Maximum expansion per OODLELZ_BLOCK_LEN is 1 byte.
Note that the compressed buffer must be allocated bigger than this (use OodleLZ_GetCompressedBufferSizeNeeded)
Discussion
convert the file name extension (max 4 chars) into a U32 for fast compares Parameters
| extension | string of an extension, not including period
|
Return Value
| return | a U32 with up to 4 chars of extension
|
Discussion
The return value is case-insensitive and does not include the dot.
| OodleLZ_GetCompressedStepForRawStep |
|
|
OO_SINTa OodleLZ_GetCompressedStepForRawStep( const void * compPtr,
OO_SINTa compAvail,
OO_SINTa startRawPos,
OO_SINTa rawSeekBytes,
OO_SINTa * pEndRawPos OODEFAULT( NULL ),
OO_BOOL * pIndependent OODEFAULT( NULL ) );Discussion
How many bytes to step a compressed pointer to advance a certain uncompressed amountParameters
Return Value
| return | the number of compressed bytes to step
|
Discussion
You should try to use GetCompressedStepForRawStep only at block granularity - both startRawPos and
rawSeekBytes should be multiples of OODLELZ_BLOCK_LEN (except at the end of the stream). As long as you
do that, then pEndRawPos will = startRawPos + rawSeekBytes.
You can use it at quantum granularity (OODLELZ_QUANTUM_LEN), but there are some caveats. You cannot step
quanta inside uncompressed blocks, only in normal LZ blocks. If you try to seek quanta inside an uncompressed
block, you will get pEndRawPos = the end of the block.
You can only resume seeking from pEndRawPos .
returns 0 for valid not-enough-data case
returns -1 for error
If compAvail is not the whole compressed buffer, then the returned step may be less than the amount you requested.
eg. if the compressed data in compAvail does not contain enough data to make a step of rawSeekBytes a smaller
step will be taken.
NOTE : can return comp step > comp avail !
Welcome to Oodle2
Oodle is a family of solutions for efficient data compression.
Oodle comes as four separate SDKs :
- Oodle Data : generic lossless data compression (Kraken, Leviathan, Mermaid, and Selkie)
- Oodle Network : packet compression to reduce bandwidth.
- Oodle Texture : encoding for BCN block-compressed GPU textures that dramatically reduces their size.
- Oodle Lossless Image : specialized encoder for lossless RGB image encoding. Faster and smaller than PNG.
Oodle Data and Network are described here. Oodle Texture and Oodle Lossless Image are available from RAD.
See About Oodle for more. Also check out the Examples.
Table of Contents
| OodleLZ_CompressScratchMemBoundType |
|
|
enum OodleLZ_CompressScratchMemBoundType
{
OodleLZ_CompressScratchMemBoundType_WorstCase = 0,
OodleLZ_CompressScratchMemBoundType_Typical = 1,
OodleLZ_CompressScratchMemBoundType_Force32 = 0x40000000
};
Discussion
Type of scratch memory bound to compute.Enumerants
Discussion
OodleLZ_CompressScratchMemBoundType_WorstCase provides a worst-case estimate for the amount
of scratch memory required for compression. If at least this much scratch memory is provided
to the compression functions, they are guaranteed to make no allocations of their own. Some
compression levels don't have simple worst-case memory use guarantees and will return
OODLELZ_SCRATCH_MEM_NO_BOUND for this type of query.
OodleLZ_CompressScratchMemBoundType_Typical returns a scratch memory estimate that will
be sufficient for most compression calls to not require additional memory, even when no
hard guarantees exist. Even so, a good estimate may not be available, in which case
OODLELZ_SCRATCH_MEM_NO_BOUND is returned.
| OodleXIOQ_OpenAndRead_Async |
|
|
OodleXHandle OodleXIOQ_OpenAndRead_Async( OodleXIOQFile * pFile,
const char * name,
void * initialReadMemory,
OO_SINTa initialReadSize,
OO_S64 initialReadPos OODEFAULT( 0 ),
OodleXFileOpenFlags fileOpenFlags OODEFAULT( OodleXFileOpenFlags_Default ),
const OodleXFileOpsVTable * vtable OODEFAULT( NULL ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
OodleXPriority priority OODEFAULT( OodleXPriority_Normal ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start opening a file for read, and do an initial readParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
If vtable is NULL, the vtable specified by the VFS mapping is used.
The file name provided is automatically run through VFS-to-OS name mapping, if applicable.
Open returns a File ref right away for your convenience, but the file is not actually open for a little while.
You can however go ahead and queue more requests on the file reference before open is complete.
You cannot call things that require an open file, such as OodleXIOQ_GetInfo.
OpenForRead is always shared access (when possible).
Also performs an initial read. Particularly useful when you need an initial header before you can start processing a file.
| Introducing the new Oodle Leviathan |
|
|
Oodle Leviathan is a new compressor that provides higher compression than Oodle Kraken while still achieving
very high decode speed.
Kraken revolutionized what was possible in lossless compression, providing insane decode speeds (3-4X faster than zlib)
with still excellent ratios. Leviathan does the same for the higher compression domain, still 2-3X faster than zlib
but with ratios that compete with or beat LZMA/7zip.
If you loved Kraken but want a little bit more compression ratio, Leviathan is for you. If you like the high
ratio of LZMA/7zip or Oodle's LZNA, but want more decode speed, Leviathan is for you.
Leviathan is way faster to decode than anything else in the world that gets as much compression. It's over 10X faster
to decode than LZMA/7zip.
For example, on the "seven" testset :
ooLeviathan7: 3.23:1 , 0.9 enc MB/s , 642.4 dec MB/s
lzmadef9 : 3.18:1 , 3.0 enc MB/s , 53.8 dec MB/s
ooKraken7 : 3.09:1 , 1.3 enc MB/s , 948.5 dec MB/s
ooMermaid7 : 2.85:1 , 2.1 enc MB/s , 1704.1 dec MB/s
zlib9 : 2.33:1 , 7.7 enc MB/s , 309.0 dec MB/s
Leviathan is currently rather slow to encode (it does extensive optimization of the bit stream), so if you're
interested in high speed encoding, Leviathan is not the answer. Kraken & Mermaid both provide fast encoding.
See for yourself how great Leviathan is by trying example_lz_chart : Example that makes a chart of OodleLZ options
Dive in to Getting Started with Oodle LZ Data Compression
Discussion
Get OodleConfigValuesParameters
| ptr | filled with OodleConfigValues
|
Discussion
Gets the current OodleConfigValues.
May be different per platform.
| FAQ: I ran out of OodleHandle table slots; what do I do? |
|
|
This is for OodleX async helpers, does not apply to Oodle Core.
Since Oodle 2.9.7, running out of handles is nearly impossible and the default maximum number of handles is much
higher. The handle table is now allocated in chunks of 2048, with the config m_num_handles_log2 setting the number
of chunks. So the default m_num_handles_log2 of 13 means 2^13 chunks of 2048, or 2^24 maximum handles total.
By default, Oodle provides 2^24 slots for async handles. This is the maximum number of handles which can be alive
at any time, but there is no limit on the number of completed or deleted handles.
If you run out of handles, it's usually due to a bug in usage that you should fix, you generally don't need
to change the size of the handle table. Some possibilities :
1. You might be leaking handles. That is, you fired off some async operation with an OodleXHandleAutoDelete
of OodleXHandleAutoDelete_No , but then you didn't delete the handle (eg. by calling
Oodle_Wait() with OodleXHandleDeleteIfDone_Yes. You should either change the lifetime to
OodleXHandleAutoDelete_Yes if you don't care about checking on the completion status of
the handle, or you should make sure to delete the handle at some point.
2. You might really need all those async handles. Simply change OodleXInitOptions:m_num_handles_log2 before
calling OodleX_Init.
| OodleXIOQ_GetInfoByName_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_GetInfoByName_AsyncDiscussion
OodleXIOQ_GetInfoByName_AsyncAndWait returns true if the file was found and info was retrieved successfully.
The return value is always false for file not found, even if you pass OodleFileNotFoundIsAnError_No.
OodleXHandle OodleXLZ_Compress_Async( OO_U32 asyncSelect,
OodleLZ_Compressor compressor,
const void * rawBuf,
OO_SINTa rawLen,
void * compBuf,
OodleLZ_CompressionLevel compressSelect,
const OodleLZ_CompressOptions * pOptions OODEFAULT( NULL ),
const void * dictionaryBase OODEFAULT( NULL ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start an async LZ compressParameters
Return Value
| return | OodleXHandle to the operation, or OodleXHandle_Null for invalid arguments
|
Discussion
Runs "wide" if asyncFlags includes OodleXAsyncSelect_Workers + OodleXAsyncSelect_Wide.
The output compressed data can be decompressed "wide" if pOptions set seekChunkReset = true.
"wide" means use many threads at once on this single operation. If the compression is selected to run wide, but the decompression
cannot run wide (eg. the compressed data does not have small independent chunks), then it will still compress wide, but on
a very large granularity, instead of the small OODLELZ_BLOCK_LEN granularity). In that case, only very large buffers will
be compressed in parallel.
Use OodleXLZ_Compress_Wait_GetResult to get the result and free associated structures.
OodleXLZ_Compress_Wait_GetResult must be called even if you don't want the result, or memory will be leaked.
| example_lz_chart : Example that makes a chart of OodleLZ options |
|
|
Discussion
Oodle example_lz_chart
The Oodle SDK comes with a pre-built exe for example_lz_chart in the bin/ directory
usage :
example_lz_chart
Run with a file name, which will be loaded and used as data to test on.
You can also toggle compile-time options with the define EXAMPLE_LZ_CHART_NUM_LEVELS below.
makes an output like this :
Oodle 2.6.3 example_lz_chart
lz_chart loading r:testsetslztestsetlzt99...
file size : 24700820
------------------------------------------------------------------------------
Selkie : super fast to encode & decode, least compression
Mermaid: fast decode with better-than-zlib compression
Kraken : good compression, fast decoding, great tradeoff!
Leviathan : very high compression, slowest decode
------------------------------------------------------------------------------
chart cell shows | raw/comp ratio : encode MB/s : decode MB/s |
All compressors run at various encoder effort levels (SuperFast - Optimal).
Many repetitions are run for accurate timing.
------------------------------------------------------------------------------
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie |1.41:675:3895|1.45:622:3888|1.53:465:3696|1.68:369:3785|1.70:342:3759|
Mermaid|1.66:436:2189|1.66:436:2188|1.79:352:2090|2.01:276:2055|2.04:261:2025|
Kraken |1.55:588:1839|1.71:419:1136|1.88:331:1087|2.10:279:1093|2.27:167:1010|
------------------------------------------------------------------------------
compression ratio (raw/comp):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 1.412 | 1.447 | 1.526 | 1.678 | 1.698 |
Mermaid| 1.660 | 1.660 | 1.793 | 2.011 | 2.041 |
Kraken | 1.548 | 1.711 | 1.877 | 2.103 | 2.268 |
------------------------------------------------------------------------------
encode speed (MB/s):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 674.548 | 621.811 | 464.555 | 369.364 | 341.588 |
Mermaid| 435.650 | 435.923 | 352.475 | 276.199 | 260.511 |
Kraken | 588.488 | 418.921 | 331.423 | 279.129 | 167.206 |
------------------------------------------------------------------------------
decode speed (MB/s):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 3894.644 | 3887.820 | 3695.984 | 3785.457 | 3758.594 |
Mermaid| 2189.030 | 2187.863 | 2090.319 | 2054.897 | 2024.692 |
Kraken | 1839.091 | 1135.920 | 1086.922 | 1093.407 | 1009.967 |
------------------------------------------------------------------------------
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie |1.75:205:3490|1.83:105:3687|1.86: 43:3815|1.93:5.1:3858|1.94:2.6:3856|
Mermaid|2.12:173:1991|2.19: 84:2177|2.21: 32:2291|2.37:2.8:2058|2.44:1.8:1978|
Kraken |2.32:112:1104|2.39: 37:1187|2.43: 20:1189|2.55:3.1:1103|2.65:1.2:1038|
Leviath|2.50: 31: 738|2.57: 17: 787|2.62:9.5: 807|2.71:1.6: 811|2.76:0.9: 776|
------------------------------------------------------------------------------
compression ratio (raw/comp):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 1.748 | 1.833 | 1.863 | 1.933 | 1.943 |
Mermaid| 2.118 | 2.194 | 2.207 | 2.367 | 2.437 |
Kraken | 2.320 | 2.390 | 2.434 | 2.551 | 2.646 |
Leviath| 2.504 | 2.572 | 2.617 | 2.707 | 2.756 |
------------------------------------------------------------------------------
encode speed (MB/s):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 204.621 | 104.758 | 42.504 | 5.102 | 2.554 |
Mermaid| 172.681 | 84.227 | 32.030 | 2.798 | 1.836 |
Kraken | 111.858 | 37.126 | 19.859 | 3.091 | 1.204 |
Leviath| 31.031 | 16.697 | 9.461 | 1.621 | 0.869 |
------------------------------------------------------------------------------
decode speed (MB/s):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 3490.442 | 3686.689 | 3814.655 | 3857.857 | 3856.226 |
Mermaid| 1991.442 | 2176.725 | 2291.498 | 2057.575 | 1977.721 |
Kraken | 1104.172 | 1186.638 | 1189.372 | 1103.148 | 1038.352 |
Leviath| 737.934 | 787.152 | 806.523 | 811.161 | 775.800 |
------------------------------------------------------------------------------
#include "../include/oodle2x.h"
#include "ooex.h"
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz_chart
#endif
//===========================================================
// NOTE : these timings are hot cache (no cache invalidation between repeats)
// that could be significant on very small buffers
struct time_and_len
{
double time;
OO_SINTa len;
};
// if scratch is NULL or insufficient, Oodle will allocate internally
static void * s_scratch_memory = NULL;
static OO_SINTa s_scratch_memory_size = 0;
static time_and_len Encode_And_Time(const void * raw_buf, OO_SINTa raw_len, void * comp_buf,
OodleLZ_Compressor compressor,
OodleLZ_CompressionLevel compression_level,
OodleLZ_CompressOptions * compression_options,
int min_repeats,
double min_total_seconds)
{
double total_seconds = 0;
time_and_len ret;
ret.time = 999999.9;
for(;;)
{
double dt = OodleX_GetSeconds();
ret.len = OodleLZ_Compress(compressor, raw_buf, raw_len, comp_buf, compression_level, compression_options,
NULL, NULL, s_scratch_memory, s_scratch_memory_size);
dt = OodleX_GetSeconds() - dt;
total_seconds += dt;
ret.time = OOEX_MIN(ret.time,dt);
min_repeats--;
if ( min_repeats <= 0 && total_seconds >= min_total_seconds )
break;
}
return ret;
}
static double Decode_And_Time( void * comp_buf, OO_SINTa comp_len, void * decode_buffer, OO_SINTa in_size,
int min_repeats,
double min_total_seconds)
{
double total_seconds = 0;
double ret = 999999.9;
for(;;)
{
double dt = OodleX_GetSeconds();
OO_SINTa decode_len = OodleLZ_Decompress( comp_buf, comp_len, decode_buffer, in_size , OodleLZ_FuzzSafe_Yes,
OodleLZ_CheckCRC_No, OodleLZ_Verbosity_None, NULL, 0, NULL,NULL, s_scratch_memory,s_scratch_memory_size );
dt = OodleX_GetSeconds() - dt;
OOEX_ASSERT_ALWAYS( decode_len == in_size );
total_seconds += dt;
ret = OOEX_MIN(ret,dt);
min_repeats--;
if ( min_repeats <= 0 && total_seconds >= min_total_seconds )
break;
}
return ret;
}
//===========================================================
static void bar()
{
OodleXLog_Printf_v1("------------------------------------------------------------------------------\n");
}
static void header(const OodleLZ_CompressionLevel * chart_levels, int num_levels)
{
//OodleXLog_Printf_v1("%5s | %-10s| %-10s| %-10s| %-10s|\n",
// "","VeryFast","Fast","Normal","Optimal1");
OodleXLog_Printf_v1("%7s|","");
for(int l=0;l<num_levels;l++)
{
OodleLZ_CompressionLevel level = chart_levels[l];
OodleXLog_Printf_v1(" %-10s|",
OodleLZ_CompressionLevel_GetName(level));
}
OodleXLog_Printf_v1("\n");
}
static const char * truncated_compressor_name( OodleLZ_Compressor compressor )
{
const char * n = OodleLZ_Compressor_GetName(compressor);
static char buf[10];
memset(buf,' ',sizeof(buf));
memcpy(buf,n,OOEX_MIN( strlen(n), sizeof(buf) ) );
buf[7] = 0;
return buf;
}
extern "C" int main(int argc,char *argv[])
{
// Init Oodle systems with default options :
OodleXInitOptions opts;
if ( ! OodleX_Init_GetDefaults(OODLE_HEADER_VERSION,&opts) )
{
fprintf(stderr,"Oodle header version mismatch.\n");
return 10;
}
// change opts here if you like
// NOTE : default options enable the OodleX thread system
// so encoders will be "Jobified"
if ( ! OodleX_Init(OODLE_HEADER_VERSION,&opts) )
{
fprintf(stderr,"OodleX_Init failed.\n");
return 10;
}
OodleXLog_Printf_v1("Oodle %s example_lz_chart <file>\n",OodleVersion);
if ( argc < 2 )
{
fprintf(stderr,"error: specify a sample data file to test on.\n");
return 10;
}
const char * in_name = argv[1];
OodleXLog_Printf_v1("lz_chart loading %s...\n",in_name);
// read the input file to the global buffer :
OO_S64 in_size_64;
void * in_buffer = OodleXIOQ_ReadMallocWholeFile_AsyncAndWait(in_name,&in_size_64);
if ( ! in_buffer)
{
OodleXLog_Printf_v0("failed to read %s\n",in_name);
return 10;
}
OodleXLog_Printf_v1("file size : " OOEX_S64_FMT "\n",in_size_64);
OO_SINTa in_size = OodleX_S64_to_SINTa_check( in_size_64 );
//-----------------------------------------------------
// test parameters :
// increase these to get more reliable timing
// decrease these to run faster
int min_encode_repeats = 2;
int min_decode_repeats = 5;
double min_total_seconds = 2.0;
#if 1
// test set of compressors :
OodleLZ_Compressor chart_compressors[] = {
OodleLZ_Compressor_Selkie, OodleLZ_Compressor_Mermaid, OodleLZ_Compressor_Kraken, OodleLZ_Compressor_Leviathan
};
#define EXAMPLE_LZ_CHART_NUM_COMPRESSORS ((int)(sizeof(chart_compressors)/sizeof(chart_compressors[0])))
OOEX_ASSERT( EXAMPLE_LZ_CHART_NUM_COMPRESSORS == 4 );
#define EXAMPLE_LZ_CHART_NUM_LEVELS 5
OodleLZ_CompressionLevel chart_levels1[] = {
OodleLZ_CompressionLevel_HyperFast4, OodleLZ_CompressionLevel_HyperFast3, OodleLZ_CompressionLevel_HyperFast2, OodleLZ_CompressionLevel_HyperFast1, OodleLZ_CompressionLevel_SuperFast,
};
OodleLZ_CompressionLevel chart_levels2[] = {
OodleLZ_CompressionLevel_VeryFast, OodleLZ_CompressionLevel_Fast, OodleLZ_CompressionLevel_Normal, OodleLZ_CompressionLevel_Optimal1, OodleLZ_CompressionLevel_Optimal3
};
OOEX_ASSERT( ((int)(sizeof(chart_levels1)/sizeof(chart_levels1[0]))) == EXAMPLE_LZ_CHART_NUM_LEVELS );
OOEX_ASSERT( ((int)(sizeof(chart_levels2)/sizeof(chart_levels2[0]))) == EXAMPLE_LZ_CHART_NUM_LEVELS );
#else
// test just 1 :
OodleLZ_Compressor chart_compressors[] = {
OodleLZ_Compressor_Kraken
};
#define EXAMPLE_LZ_CHART_NUM_COMPRESSORS 1
#define EXAMPLE_LZ_CHART_NUM_LEVELS 1
OodleLZ_CompressionLevel chart_levels1[] = {
OodleLZ_CompressionLevel_None
};
OodleLZ_CompressionLevel chart_levels2[] = {
OodleLZ_CompressionLevel_Optimal2
};
#endif
//===========================================================
OO_SINTa comp_lens[EXAMPLE_LZ_CHART_NUM_COMPRESSORS][EXAMPLE_LZ_CHART_NUM_LEVELS];
double decode_speeds[EXAMPLE_LZ_CHART_NUM_COMPRESSORS][EXAMPLE_LZ_CHART_NUM_LEVELS];
double encode_speeds[EXAMPLE_LZ_CHART_NUM_COMPRESSORS][EXAMPLE_LZ_CHART_NUM_LEVELS];
//-----------------------------------------------------
void * comp_buf;
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Invalid,in_size);
comp_buf = OodleXMallocBig(comp_buf_size);
void * decode_buffer = OodleXMallocBig( in_size );
//-----------------------------------------------------
// allocate scratch
// get enough so the decoder won't allocate
// and encoders up to Leviathan-Normal won't either
// but optimals will
s_scratch_memory_size = OodleLZ_GetCompressScratchMemBound(OodleLZ_Compressor_Leviathan,OodleLZ_CompressionLevel_Normal,in_size,NULL);
s_scratch_memory_size = OOEX_MAX( s_scratch_memory_size , OodleLZDecoder_MemorySizeNeeded() );
s_scratch_memory = OodleXMalloc(s_scratch_memory_size);
//-----------------------------------------------------
bar();
OodleXLog_Printf_v1("Selkie : super fast to encode & decode, least compression\n");
OodleXLog_Printf_v1("Mermaid: fast decode with better-than-zlib compression\n");
OodleXLog_Printf_v1("Kraken : good compression, fast decoding, great tradeoff!\n");
OodleXLog_Printf_v1("Leviathan : very high compression, slowest decode\n");
bar();
OodleXLog_Printf_v1("chart cell shows | raw/comp ratio : encode MB/s : decode MB/s | \n");
OodleXLog_Printf_v1("All compressors run at various encoder effort levels (SuperFast - Optimal).\n");
OodleXLog_Printf_v1("Many repetitions are run for accurate timing.\n");
bar();
for(int twice=0;twice<2;twice++)
{
const OodleLZ_CompressionLevel * chart_levels =
twice ? chart_levels2 : chart_levels1;
int num_compressors = EXAMPLE_LZ_CHART_NUM_COMPRESSORS;
// don't bother running Leviathan at the HyperFast modes :
if ( ! twice && chart_compressors[num_compressors-1] == OodleLZ_Compressor_Leviathan ) num_compressors--;
header(chart_levels,EXAMPLE_LZ_CHART_NUM_LEVELS);
for(int c=0;c<num_compressors;c++)
{
OodleLZ_Compressor compressor = chart_compressors[c];
OodleXLog_Printf_v1("%s|",truncated_compressor_name(compressor));
// iterate backwards so we start the slowest first
for(int l=0;l<EXAMPLE_LZ_CHART_NUM_LEVELS;l++)
{
OodleLZ_CompressionLevel level = chart_levels[l];
time_and_len tl =
Encode_And_Time(in_buffer,in_size,comp_buf,compressor,level,NULL,min_encode_repeats,min_total_seconds);
OO_SINTa comp_len = tl.len;
double encode_seconds = tl.time;
double encode_mbps = (in_size/1000000.0) / encode_seconds;
encode_speeds[c][l] = encode_mbps;
comp_lens[c][l] = comp_len;
double ratio = (double)in_size/comp_len;
if ( ratio >= 10.0 )
OodleXLog_Printf_v1("%4.1f",ratio);
else
OodleXLog_Printf_v1("%4.2f",ratio);
if ( encode_mbps >= 10.0 )
OodleXLog_Printf_v1(":%3d:",(int)(encode_mbps+0.5));
else
OodleXLog_Printf_v1(":%3.1f:",encode_mbps);
double decode_seconds = Decode_And_Time( comp_buf, comp_len, decode_buffer, in_size ,
min_decode_repeats,min_total_seconds);
OOEX_ASSERT_ALWAYS( memcmp(decode_buffer,in_buffer,in_size) == 0 );
double decode_mbps = (in_size/1000000.0) / decode_seconds;
decode_speeds[c][l] = decode_mbps;
OodleXLog_Printf_v1("%4d|",(int)(decode_mbps+0.5));
}
OodleXLog_Printf_v1("\n");
}
//-----------------------------------------------------
bar();
OodleXLog_Printf_v1("compression ratio (raw/comp):\n");
header(chart_levels,EXAMPLE_LZ_CHART_NUM_LEVELS);
for(int c=0;c<num_compressors;c++)
{
OodleLZ_Compressor compressor = chart_compressors[c];
OodleXLog_Printf_v1("%s|",truncated_compressor_name(compressor));
// iterate backwards so we start the slowest first
for(int l=0;l<EXAMPLE_LZ_CHART_NUM_LEVELS;l++)
{
double ratio = (double)in_size/comp_lens[c][l];
OodleXLog_Printf_v1("%9.3f |",ratio);
}
OodleXLog_Printf_v1("\n");
}
bar();
OodleXLog_Printf_v1("encode speed (MB/s):\n");
header(chart_levels,EXAMPLE_LZ_CHART_NUM_LEVELS);
for(int c=0;c<num_compressors;c++)
{
OodleLZ_Compressor compressor = chart_compressors[c];
OodleXLog_Printf_v1("%s|",truncated_compressor_name(compressor));
// iterate backwards so we start the slowest first
for(int l=0;l<EXAMPLE_LZ_CHART_NUM_LEVELS;l++)
{
OodleXLog_Printf_v1("%11.3f |",encode_speeds[c][l]);
}
OodleXLog_Printf_v1("\n");
}
bar();
OodleXLog_Printf_v1("decode speed (MB/s):\n");
header(chart_levels,EXAMPLE_LZ_CHART_NUM_LEVELS);
for(int c=0;c<num_compressors;c++)
{
OodleLZ_Compressor compressor = chart_compressors[c];
OodleXLog_Printf_v1("%s|",truncated_compressor_name(compressor));
// iterate backwards so we start the slowest first
for(int l=0;l<EXAMPLE_LZ_CHART_NUM_LEVELS;l++)
{
OodleXLog_Printf_v1("%11.3f |",decode_speeds[c][l]);
}
OodleXLog_Printf_v1("\n");
}
bar();
} // twice
//-----------------------------------------------------
OodleXFree(s_scratch_memory);
OodleXFreeBig(decode_buffer);
OodleXFreeBig(comp_buf);
OodleXFree_IOAligned(in_buffer);
//OodleX_Shutdown();
OodleX_Shutdown(NULL,OodleX_Shutdown_LogLeaks_Yes,0);
//OodleXLog_Printf_v1("press a key\n");
//fgetc(stdin);
return 0;
}
//=================================================
Discussion
Start a high level IO request to copy a fileParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Copy a file as a single IOQ op.
NOTE : you generally do not want this. Use Oodle_CopyFile_Async instead. Using this call blocks
the IOQ from servicing streams or doing other work.
CopyFile is a single IOQ op so it is guaranteed to be done before a subsequent call to
OodleIOQ_OpenForRead on the to file, so it is useful for async transparent mirroring.
(the same is not true of Oodle_CopyFile_Async which has undefined scheduling).
About OodleIOQ
OodleIOQ is the lowest level async IO layer that the client should use.
(OodleXFileOpsVTable is lower level; the IOQ uses it to do its file ops,
but the client should not call OodleXFileOpsVTable functions directly)
Every file IO operation on an OodleIOQ file is completely asynchronous, they
never block the calling thread. Calls like OodleXIOQ_CloseFile_Async just start an
operation; you can then OodleX_Wait on the returned handle if you like.
(there are convenience wrappers provided (like OodleXIOQ_WriteWholeFile_AsyncAndWait)
which just start an async op and then wait on it).
An OodleXIOQFile does not have a file pointer (position) or buffer associated with it.
They are stateless. Multiple threads may be using the same OodleXIOQFile at the same time.
Many people are initially confused by the fact that OodleXIOQ_OpenForRead_Async is
asynchronous. That means that if you try to open a file that doesn't exist, the call
to OodleXIOQ_OpenForRead_Async will still succeed ; the success of that call only means
that it was successful in starting the operation. If you want to know if the operation
actually succeeded, you must get the status of the OodleXHandle returned.
OodleXIOQ_OpenForRead_Async will always create an OodleXIOQFile, but it may not contain
a valid disk file until/unless the operation succeeds.
Every operation you start on the IOQ has a corresponding OodleXHandle. That handle
can be used to check status (OodleX_GetStatus) or wait on completion. If you don't need
to track the progress of an IO operation, you can make the handle have a autoDelete of
OodleXHandleAutoDelete_Yes, and just ignore the handle.
IO Operation Ordering
IOQ operations are implicitly FIFO in the absense of other user-provided scheduling
constraints. The IOQ may transparently reorder operations when that reordering cannot
affect their behavior; for example Reads and OpensForRead can be moved against each
other, but Reads will never be moved across Writes.
FIFO ordering is very handy for the client to batch up chains of operations and know
what order they will be done in.
For example you can do
OodleXIOQ_Read_Async(file1,buffer,sizeof(buffer),0);
OodleXIOQ_Write_Async(file2,buffer,sizeof(buffer),0);
this will enqueue a read from file1 into "buffer" and then a write to file2
from "buffer". Because of the implicit FIFO order, this will do what you expect -
the read will complete and fill buffer before I try to write from it. Without
the implicit FIFO those operations would have to be explicitly ordered by dependencies.
Every OodleIOQ function that starts an async operation optionally takes a list of
dependencies. If you pass dependencies, then that operation will not participate in
the FIFO sequence; its ordering is set by the completion of the dependencies.
Alignment and Buffering
The Oodle IOQ can be used with aligned or unaligned IO. Unaligned IO is faster on some
platforms, and sometimes has lower memory use, so it should be used when possible.
Some platforms can only do true async IO on sector-aligned boundaries, so if you use unaligned
IO on those platforms the IO thread will be running synchronous IO (it will still appear
asynchronous to the client code using the IOQ).
If you don't want to use aligned IO, open all files with OodleXFileOpenFlags_Buffered. Then
the file sizes and positions you pass to IOQ can be arbitrary.
If you wish to use aligned IO, all pointers and file positions that are used in OodleIOQ
calls must be aligned. The easiest way is to align everything to OODLEX_IO_MAX_ALIGNMENT,
which works on any platform. To do this you can allocate your
buffers using OodleXMalloc_IOAligned (or you can always align pointers yourself).
If you don't want to use OODLEX_IO_MAX_ALIGNMENT, OodleXIOQ_GetInfo will report the actual
alignment required for a given file on that platform. Note that the alignment may differ
on different devices, so it should not be stored as a global.
To do non-aligned IO when you have opened your files in aligned mode, you have a few options.
To read an unaligned part of a file, simply
move the start position back to the previous alignment, with OodleX_IOAlignDownS64 and move
the end position up with OodleX_IOAlignUpS64 , then read this larger block and you will find
your unaligned piece inside. Alternatively, OodleXIOQ_ReadUnalignedAdjustPointer_Async
is a helper which does this for you.
Another option is to use the higher level OodleFile
routines, which do large aligned IO's to fill a buffer and let you grab arbitrary pieces of
that buffer. You could also use an OodleIOQStream if you are always reading forward
linearly.
To write an unaligned portion of a file (on a file opened in aligned-IO mode),
it's probably best to use the higher level OodleFile
interface which does synchronous buffering for you. Otherwise you have to first read the
surrounding larger aligned chunks, replace your unaligned portion, and write it back out again.
Tracking Errors
You can track the error status and completion of each request using the OodleXHandle it
returns.
Alternatively, OodleIOQ supports a more "file-first" mode of usage.
OodleXIOQFile tracks the last error of any request on it, so you can just fire off a
bunch of requests and then check OodleXIOQ_GetLastError on the file to see if any of
them failed.
OodleXIOQFile also tracks the last handle of any operation started on it, so you can
use OodleIOQ_GetLastRequest or OodleXIOQ_WaitLastRequest to check if all ops on a file
are done.
With this mode of usage, you probably want to start the operations with a autoDelete of
OodleXHandleAutoDelete_Yes so that you don't have to delete them manually.
Threading Semantics
IOQ operations are all inherently atomic and thread safe. However, your user objects which
you pass to the IOQ calls must be managed by you.
Buffers used in IOQ calls must be kept allocated for the lifetime of the operation.
eg. if you call OodleXIOQ_Read_Async , the memory pointer you pass in must be kept allocated
until the handle completes. Don't use memory on the stack if your function can return
before the operation is complete.
Strings passed into IOQ requests (eg. file names) are copied and can be freed immediately
after starting the operation.
Discussion
Value 0 of Jobify handles is reserved to mean none
* Wait(OODLE_JOB_NULL_HANDLE) is a nop
* if RunJob returns OODLE_JOB_NULL_HANDLE it means the job
* was run synchronously and no wait is required
| OodleNet_Plugins_SetAssertion |
|
|
Discussion
Install the callback used by Oodle Core for assertsParameters
Return Value
| return | returns the previous function pointer
|
Discussion
Use this function to install your own display for Oodle Core assertions.
This will only happen in debug builds.
The default implementation in debug builds, if you install nothing, uses the C stderr printf for logging,
except on Microsoft platforms where it uses OutputDebugString.
WARNING : this function is NOT thread safe! It should be done only once and done in a place where the caller can guarantee thread safety.
| OODLE_ALLOW_DEPRECATED_COMPRESSORS |
|
|
Discussion
If you need to encode with the deprecated compressors, define this before including oodle2.hDiscussion
You may still decode with them without defining this.
| FAQ: What is SINTa? How do I load files bigger than 2 GB? |
|
|
Oodle uses 64 bit integers for file sizes and file positions. This allows you to manipulate
files that are larger than 2 GB (31 bits) even on 32 bit systems.
Oodle uses a variable called "SINTa" for most memory buffer sizes. The "a" stands for address
and indicates it is an integer the size of an address (eg. 32 bits on 32 bit systems and
64 bits if pointers are 64 bits). It's the same idea as "intptr_t" or "ptrdiff_t" in standard C.
Many of the Oodle APIs, particularly the higher level ones that are provided for convenience,
only work if they can load an entire file into memory (for example, things like OodleXIOQ_ReadMallocWholeFile_Async).
On 64 bit systems, everything is simple - SINTa is the same as S64 and you can make buffers big enough
to load entire files.
On 32 bit systems, SINTa is S32, and not all file sizes can fit in an SINTa.
Whenever you have to cast an S64 to an SINTa, the safest way is to use OodleX_S64_to_SINTa_check ; on 64 bit systems
this is a no-op, but on 32 bit systems it emits an error if the size is too large.
If you want to write code that can handle files larger than 2 GB on a 32 bit system, you can do this with Oodle,
you simply have to use the lower level APIs and avoid the higher level helpers that deal with whole file buffers.
For example the basic OodleXIOQ_Read_Async that all other Oodle reads is built on can be used to read small parts of a large file.
Similarly the whole buffer decompression routines like OodleLZ_Decompress cannot be used if the whole buffer doesn't fit in
memory, but you can use the incremental decompression (OodleLZDecoder_DecodeSome) to stream a larger-than-memory file through
memory and decompress it.
Oodle never makes match references further away than 1 GB, so you can encode huge buffers in 64-bit and they are gauranteed to
be decodeable in 32-bit.
Oodle does not currently provide compression routines that can handle larger than memory buffers. But you can simply compress
part of the file independently and concatenate together the chunks, because OodleLZ compressed data is binary concatenatable.
(see About OodleLZ). For example you could encode 0.5 GB per call, with 0.5 GB of overlap to the previous chunk. The data you
emit this way could be decoded in one call on 64 bit systems, or with multiple calls on 32 bit.
See the "oozi" example for compression of very large files.
NOTE : I highly recommend using Oodle in 64 bit. Not only will it avoid memory limitations, it is significantly faster.
| OodleXIOQ_CloseFileRename_Async |
|
|
Discussion
Start a close-file-rename requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Does an OodleXIOQ_CloseFile_Async , then renames the file to "renameTo" - but only if there were no errors in writing
the file. To stop unimportant errors from causing OodleXIOQ_CloseFileRename_Async to fail, use OodleXIOQ_ClearError
before calling this.
renameTo can be NULL to cancel the close and delete the temp file.
Useful with OodleXIOQ_OpenForWriteTempName_Async.
ozip is utility for Oodle that's designed to act like gzip, for piping compresed streams on Unix-like platforms.
ozip comes with Oodle as a compiled executable (in the bin/ directory), and is also available as source code.
The ozip code is public domain and is provided on github :
https://github.com/jamesbloom/ozip/
Oodle is not public domain and cannot be used without an Oodle license. The compiled ozip executable with Oodle
is not public domain.
ozip can be used to test Oodle without having to write any code.
ozip -k --kraken -6 lzt99
makes lzt99.ooz with Kraken at level 6 (Optimal2)
When working on pipes, ozip incrementally flushes data so the stream keeps moving and memory use is limited.
It can be used as a drop in to most programs that expect gzip-like pipes. For example with tar :
tar -cf archive.tar.ooz -I 'ozip --fast' subdir
tar cf - subdir | ozip --best > archive.tar.gz
ozip can also be used for benchmarking. In benchmark mode it makes no output and runs the operation many times
to get accurate timing. You can use this to benchmark specific files with specific options, unlike example_lz_chart : Example that makes a chart of OodleLZ options
which runs a matrix of compressors and levels with default options.
For example :
ozip -b lzt02
K 4 lzt02 : 755121 -> 161674 (4.671), 57.6 MB/s, 1808.7 MB/s
ozip -b lzt02 -9 --kraken
K 9 lzt02 : 755121 -> 135523 (5.572), 1.9 MB/s, 1547.4 MB/s
ozip -b lzt02 -9 --hydra
H 9 lzt02 : 755121 -> 130035 (5.807), 0.6 MB/s, 1291.0 MB/s
ozip -b lzt02 -9 --hydra -os512
H 9 lzt02 : 755121 -> 141414 (5.340), 0.6 MB/s, 1818.3 MB/s
zstd -b22 lzt02
22#lzt02 : 755121 -> 197827 (3.817), 10.21 MB/s , 815.0 MB/s
See some tips on benchmarking here : Tips for benchmarking a compressor
| OodleXLZ_ReadAndDecompress_Wide_Async |
|
|
OodleXHandle OodleXLZ_ReadAndDecompress_Wide_Async( OO_U32 asyncSelect,
const OodleLZ_SeekTable * seekTable,
const void * packedDataPtr,
OO_SINTa packedLen,
OO_SINTa packedLenPreviouslyRead,
OodleXIOQFile packedFile,
OO_S64 packedDataStartPos,
void * rawArray,
OO_SINTa rawArrayLen,
OodleLZ_FuzzSafe fuzzSafe,
OodleLZ_CheckCRC checkCRC,
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OodleLZ_PackedRawOverlap packedRawOverlap OODEFAULT( OodleLZ_PackedRawOverlap_No ),
OodleXIOQFile writeToFile OODEFAULT( 0 ),
OO_S64 writeToFileStartPos OODEFAULT( 0 ),
OodleXHandle * pWriteHandleGroup OODEFAULT( 0 ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start an async LZ decompress, possibly read packed data and write raw dataParameters
| asyncSelect | logical OR of OodleXAsyncSelect flags determine how the async is run
|
| seekTable | seek locations as created by OodleLZ_CreateSeekTable
|
| packedDataPtr | pointer to LZ compressed data
|
| packedLen | compressed data length
|
| packedLenPreviouslyRead | number of packed bytes already in packedDataPtr from previous IO ; eg. packedLen if the whole buffer is full
|
| packedFile | OodleXIOQFile to read packed bytes from
|
| packedDataStartPos | file position where the packed data starts (must be misaligned the same way as packedDataPtr)
|
| rawArray | pointer to memory filled with decompressed data
|
| rawArrayLen | length of decompressed data
|
| checkCRC | if OodleLZ_CheckCRC_Yes, the decompressor checks the crc to ensure data integrity
|
| verbosity | (optional) if not OodleLZ_Verbosity_None, will log some information
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder.
|
| decBufSize | (optional) size of circular buffer starting at decBufBase
|
| packedRawOverlap | (optional) if OodleLZ_PackedRawOverlap_Yes, the compressed data is in the same memory array as the output raw data
|
| writeToFile | (optional) OodleXIOQFile to write raw data to
|
| writeToFileStartPos | (optional) file position where writeToFile should start (must be OODLEX_IO_MAX_ALIGNMENT aligned)
|
| pWriteHandleGroup | (optional) if writeToFile is given, this is filled with an OodleAsyncGroup OodleXHandle containing all the file IO operations
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | OodleXHandle to the operation, or OodleXHandle_Null for invalid arguments
|
Discussion
Start an async LZ decompress with the runner specified in asyncSelect.
The entire Read-Decomp-Write is done with maximum parallelism; reads are done in chunks, as each compressed chunk is available,
it is decompressed, and as raw chunks are done, they are written.
A note on the alignment of packedDataPtr and packedDataStartPos : the simplest way is if both are OODLEX_IO_MAX_ALIGNMENT.
However, if the packed data starts some non-aligned way into the file, ensure the misalignment of both is the same. This is
automatic if you allocate a buffer to correspond to the whole file, or start your read at the preceding aligned position.
If you have the data already read into memory, use OodleXLZ_Decompress_Wide_Async instead.
To use OodleLZ_PackedRawOverlap_Yes , make a buffer of size at least OodleLZ_GetInPlaceDecodeBufferSize ; you then read
the compressed data in to the end of that array, and decompress with the raw pointer set to the front of that array.
This lets you avoid allocating two large arrays. It does hurt parallelism.
rawArray and packedDataPtr memory blocks passed to this function must be kept alive for the duration of the async.
To use this function, you should have stored the seekTable for the compressed data in a file.
NOTE !! : if writeToFile is provided, the writes are async and are NOT necessarily done when
the returned handle is done; the returned handle is for the decompress. They are done when the
handle in pWriteHandleGroup is done. You must not free the buffer being written until *pWriteHandleGroup is done.
Discussion
Change the default file ops vtableDiscussion
pNewVTable is copied into the global default file ops.
These file ops are used by Oodle whenever no other vtable is provided.
WARNING : access to OodleXFileOpsVTable is not thread safe. It should generally only be done
at app initialization time to set your desired func pointers, and then not done thereafter.
| OodleLZ_GetDecodeBufferSize |
|
|
Discussion
Get the size you must malloc the decode (raw) bufferParameters
| compressor | compressor used; OodleLZ_Compressor_Invalid to make it enough for any compressor
|
| rawSize | uncompressed (raw) size without padding
|
| corruptionPossible | true if it is possible for the decoder to get corrupted data
|
Return Value
| return | size of buffer to malloc; slightly larger than rawSize if padding is needed
|
Discussion
As of Oodle 2.9.0 this function is deprecated. For all new codecs you can just use the size of the
uncompressed data for the decode buffer size (rawSize), no padding is needed.
Note that LZB16 is still supported in 2.9.0 but does require padding when used in a circular
window (which is deprecated).
This padding is necessary for the older compressors when FuzzSafe_No is used. The old compressors
and FuzzSafe_No are no longer supported.
If corruptionPossible is true, a slightly larger buffer size is returned.
If corruptionPossible is false, then you must ensure that the decoder does not get corrupted data,
either by passing OodleLZ_CheckCRC_Yes , or by your own mechanism.
Note about possible overrun in LZ decoding (applies to the old non-fuzz-safe compressors) :
as long as the compresseddata is not corrupted,
and you decode either the entire compressed buffer, or an integer number of "seek chunks" (OODLELZ_BLOCK_LEN),
then there will be no overrun. So you can decode LZ data in place and it won't stomp any following bytes.
If those conditions are not true (eg. decoding only part of a larger compressed stream, decoding
around a circular window, decoding data that may be corrupted), then there may be some limited amount of
overrun on decode, as returned by OodleLZ_GetDecodeBufferSize.
Discussion
Align down to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x down to OODLEX_IO_MAX_ALIGNMENT
| t_fp_OodleCore_Plugin_WaitJob |
|
|
Discussion
Function pointer type for OodleCore_Plugins_SetJobSystemParameters
| job_handle | a job handle returned from RunJob. Never 0.
|
| user_ptr | is passed through from the OodleLZ_CompressOptions.
|
Discussion
Waits until the job specified by job_handle is done and cleans up any associated resources. Oodle
will call WaitJob exactly once for every RunJob call that didn't return 0.
If job_handle was already completed, this should clean it up without waiting.
A handle value should not be reused by another RunJob until WaitJob has been done with that value.
WaitJob will not be called from running jobs. It will be only be called from the original thread that
invoked Oodle. If you are running Oodle from a worker thread, ensure that that thread is allowed to wait
on other job threads.
See About Oodle Job Threading Plugins
enum OodleXPriority
{
OodleXPriority_Normal = 2,
OodleXPriority_Default = OodleXPriority_Normal,
OodleXPriority_NoPopOnWait = 1,
OodleXPriority_Force32 = 0x40000000
};
Discussion
Priority for async tasks. Higher prority is done before lower priority.
Async work is (on average) FIFO within priority group.Enumerants
Discussion
Core Jobs from Jobify are above priority Normal.
"low priority" work is "high level", large groups of actions
"high priority" work is "low level", splitting small actions into micro actions that should complete soon
OodleNetwork1 compressor for shared-dictionary network channel compression.
Discussion
User-provided callback for decompressionParameters
| userdata | the data you passed for pcbData
|
| rawBuf | the decompressed buffer
|
| rawLen | the total decompressed length
|
| compBuf | the compressed buffer
|
| compBufferSize | the total compressed length
|
| rawDone | number of bytes in rawBuf decompressed so far
|
| compUsed | number of bytes in compBuf consumed so far
|
Discussion
OodleDecompressCallback is called incrementally during decompression.
| example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core |
|
|
Discussion
Oodle example_lz_simple
Very simple example of OodleLZ memory -> memory compression & decompression.
Uses stdio for file IO to load an input file.
See example_lz : Example demonstrating LZ compression and decompression for more advanced OodleLZ usage.
example_lz_simple only uses Oodle Core, no Oodle Ext
include oodle2.h and not oodle2x.h
#include "../include/oodle2.h"
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz_simple
#endif
#include "read_whole_file.h"
extern "C" int main(int argc,char *argv[])
{
No initialization is needed for Oodle2 Core
we let Oodle Core use the default system plugins (the C stdlib)
To change them, use Core plugins
// optional check to make sure header matches lib :
if ( ! Oodle_CheckVersion(OODLE_HEADER_VERSION) )
{
fprintf(stderr,"Oodle header version mismatch\n");
return 10;
}
// get args :
const char * in_name;
if ( argc < 2 )
{
in_name = "r:\\testsets\\lztestset\\lzt02";
}
else
{
in_name = argv[1];
}
read input file using stdio
OO_SINTa length;
void * buf = read_whole_file(in_name,&length);
if ( ! buf )
{
fprintf(stderr,"couldn't open : %s\n",in_name);
return 10;
}
Run OodleLZ_Compress from memory (buf) to memory (compbuf)
Use the OodleLZ_Compressor_Kraken compressor. Kraken is an amazing balance of good compression and fast decode
speed. It should generally be your first choice, then try Mermaid or Leviathan if you want faster decodes or more
compression.
Use OodleLZ_CompressionLevel_Normal level of effort in the encoder. Normal is a balance of encode speed and compression
ratio. Different levels trade off faster or slower encoding for compressed size. See OodleLZ_CompressionLevel.
See About OodleLZ for information on selection of the compression options.
This call is synchronous and not threaded; see example_lz : Example demonstrating LZ compression and decompression for an example using the async compression APIs.
OodleLZ_Compressor compressor = OodleLZ_Compressor_Kraken;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Normal;
//OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Optimal; // for high compression, slower encode
// allocate memory big enough for compressed data :
void * compbuf = malloc( OodleLZ_GetCompressedBufferSizeNeeded(compressor,length) + sizeof(length) );
if ( compbuf == NULL )
return 10;
char * compptr = (char *)compbuf;
memcpy(compptr,&length,sizeof(length));
compptr += sizeof(length);
// compress :
OO_SINTa complen = OodleLZ_Compress(compressor,buf,length,compptr,level);
compptr += complen;
// log about it :
// full compressed size also includes the header +(int)sizeof(length)
printf("%s compressed %d -> %d\n",in_name,(int)length,(int)complen);
Run OodleLZ_Decompress from memory (compbuf) to memory (decbuf)
Note that you must provide the exact decompressed size. OodleLZ data is headerless; store the size in
your own header.
We will allocate the needed decoder scratch mem to pass in, then OodleLZ_Decompress will do no allocations
internally. In real use you might want to keep the scratch mem allocated across calls so it doesn't need to
be allocated and freed each time. The scratch mem can be reused, but cannot be used by multiple threads at the
same time. See also example_lz_noallocs
OO_SINTa declength;
compptr = (char *)compbuf;
memcpy(&declength,compptr,sizeof(declength));
compptr += sizeof(declength);
assert( length == declength );
// malloc for decompressed buffer :
void * decbuf = malloc( declength );
// allocate the decoder scratch memory needed
// if we pass NULL for these (the default argument)
// then OodleLZ_Decompress will allocate them internally
OO_SINTa decoderScratchMemSize = OodleLZDecoder_MemorySizeNeeded(compressor,-1);
void * decoderScratchMem = malloc( decoderScratchMemSize );
// do the decompress :
OO_SINTa decompress_return = OodleLZ_Decompress(compptr,complen,decbuf,declength,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,NULL,0,NULL,NULL,
decoderScratchMem,decoderScratchMemSize);
// check it was successful :
assert( decompress_return == length );
assert( memcmp(buf,decbuf,length) == 0 );
if ( decompress_return != length )
return 10;
printf("decompressed successfully.\n");
And finish up. No shutdown is needed for Oodle2 Core.
// free all the memory :
free(buf);
free(compbuf);
free(decbuf);
free(decoderScratchMem);
return 0;
}
Discussion
Block the calling thread until none of the provided handles are PendingParameters
Return Value
Discussion
Blocks until ALL handles are done.
Returns OodleXStatus_Error if any of the handles in the array is done with status OodleXStatus_Error.
Discussion
OodleLZ_FuzzSafe (deprecated)Discussion
About fuzz safety:
Fuzz Safe decodes will not crash on corrupt data. They may or may not return failure, and produce garbage output.
Fuzz safe decodes will not read out of bounds. They won't put data on the stack or previously in memory
into the output buffer.
As of Oodle 2.9.0 all compressors supported are fuzzsafe, so OodleLZ_FuzzSafe_Yes should always be used and this
enum is deprecated.
| OodleNetwork1_SelectDictionaryFromPackets |
|
|
OO_BOOL OodleNetwork1_SelectDictionaryFromPackets( void * dictionary_to_fill,
OO_S32 dictionary_size,
OO_S32 htbits,
const void * * dictionary_packet_pointers,
const OO_S32 * dictionary_packet_sizes,
OO_S32 num_dictionary_packets,
const void * * test_packet_pointers,
const OO_S32 * test_packet_sizes,
OO_S32 num_test_packets );Discussion
Build a dictionary for OodleNetwork1 from packetsParameters
Return Value
| return | true on success, false otherwise.
|
Discussion
Fills out a dictionary from packets by rating the packets and choosing the most useful.
Can be used with OodleNetwork1 UDP or TCP.
The dictionary_packet_pointers are used to fill the dictionary. The test_packet_pointers
are used to evaluate the quality of the dictionary. They should not be the same packets!
They should both be independent random representative samples of your network traffic.
The packets used here should not be the same ones used for training! (eg. with
OodleNetwork1UDP_Train or OodleNetwork1TCP_Train).
This function is to be called before OodleNetwork1_Shared_SetWindow.
This function may take a lot of time if the test_packet_pointers set is too large. It should
be large enough to provide good test ratings, but too large slows it down for no reason.
Something like 50 MB of packet data is usually sufficient.
See TestOodleNetwork1PacketCoder_SelectDictionaryAndTrain in example_packet : Example demonstrating network packet compression
On platforms where OodleNetwork1_SelectDictionarySupported returns false, this
function is not supported.
It's advised to run this function in 64-bit, as it can use a lot of memory.
Discussion
Return value of OodleLZ_Decompress on failure
Discussion
Opaque type for OodleLZDecoderDiscussion
See OodleLZDecoder_Create
Discussion
Logs an OS error code with a detailed text messageParameters
Discussion
Calls OodleXLog_Printf to output a detailed error, as created by OodleXIOQ_GetErrorDetails.
| OodleNetwork1TCP_State_InitAsCopy |
|
|
Discussion
Initialize a OodleNetwork1TCP_State as a copy of another stateParameters
Discussion
Resets state to a copy of from state.
Generally from was made with OodleNetwork1TCP_Train and then saved to disk so that it could be stored on both the client and server.
Discussion
Align up to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x up to OODLEX_IO_MAX_ALIGNMENT
enum OodleXHandleKickDelayed
{
OodleXHandleKickDelayed_No = 0,
OodleXHandleKickDelayed_Yes = 1,
OodleXHandleKickDelayed_Force32 = 0x40000000
};
Discussion
Normally async tasks are run as soon as possible; sometimes when spawning many tasks, you might not
want to let the thread switch immediately, so it can be better to fire several tasks with OodleXHandle_KickDelayed
and then kick them all together. ("kick" means activate worker threads to do the tasks)
Enumerants
| OodleXIOQ_GetInfoByName_GetResult |
|
|
Discussion
Finish an asynchronous GetInfo requestParameters
Return Value
| return | OodleXStatus of the request
|
Discussion
pInfo is filled if the OodleXStatus returned is OodleXStatus_Done. If the return value is something
else, pInfo is untouched (eg. not invalidated!).
If the file does not exist and OodleFileNotFoundIsAnError_No was passed, this function will return
OodleXStatus_Done but pInfo will be set to a OodleXFileInfo:size of OODLEX_FILE_SIZE_INVALID;
Oodle is as uniform as possible across platforms.
The contents of the distribution, and how to build and use it on that platorm is described here.
| OodleXIOQ_ReadUnalignedAdjustPointer_Async |
|
|
Discussion
Start a read request with unaligned position or sizeParameters
| pPtr | filled with a pointer to the read memory; NULL if the read is impossible
|
| file | the file to act on
|
| memory | memory to read into (no alignment required)
|
| readSize | number of bytes to read (no alignment required)
|
| position | file position to start the read (no alignment required)
|
| memorySize | the size of the buffer at "memory"
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| priority | (optional) priority of the operation ; see OodleXPriority
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
memorySize should be larger than size ; generally at least aligned up with OodleX_IOAlignUpS64.
OodleXIOQ_ReadUnalignedAdjustPointer_Async reads a larger chunk than [position,size] , aligning down the start
and aligning up the end. It reads somewhere into [memory,memorySize]. The returned pointer is somewhere
in memory and contains the bytes you wanted from _position.
If memorySize is not big enough, it returns NULL.
| OodleXIOQ_MakeAllDirs_Async |
|
|
Discussion
Start a high level IO request to make all dirs in nameParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Makes the dirs in name in sequence. name can be a file name, or a path with trailing path delim.
eg. if name is "a/b/c/d" then dir a is made, then b, then c, but not d.
OodleLZ low level synchronous memory to memory lossless data compression.
- Defines
- Enumerants
- Structures
- Functions
- Typedefs
Discussion
Set OodleConfigValuesParameters
| ptr | your desired OodleConfigValues
|
Discussion
Sets the global OodleConfigValues from your struct.
You should call Oodle_GetConfigValues to fill the struct, then change the values you
want to change, then call Oodle_SetConfigValues.
This should generally be done before doing anything with Oodle (eg. even before OodleX_Init).
Changing OodleConfigValues after Oodle has started has undefined effects.
| OodleXLZ_Decompress_Narrow_Async |
|
|
OodleXHandle OodleXLZ_Decompress_Narrow_Async( OO_U32 asyncSelect,
const void * packedDataPtr,
OO_SINTa packedLen,
void * rawPtr,
OO_SINTa rawChunkLen,
OodleLZ_FuzzSafe fuzzSafe OODEFAULT( OodleLZ_FuzzSafe_No ),
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OodleDecompressCallback * pcb OODEFAULT( NULL ),
void * pcbData OODEFAULT( NULL ),
void * decMem OODEFAULT( NULL ),
OO_SINTa decMemSize OODEFAULT( 0 ),
OodleLZ_Decode_ThreadPhase threadPhase OODEFAULT( OodleLZ_Decode_Unthreaded ),
OodleXIOQFile writeToFile OODEFAULT( 0 ),
OO_S64 writeToFileStartPos OODEFAULT( 0 ),
OodleXHandle writeHandleGroup OODEFAULT( 0 ),
OO_S32 writeHandleGroupIndex OODEFAULT( 0 ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ),
OodleXPriority work_priority OODEFAULT( OodleXPriority_Normal ) );Discussion
Start an async LZ decompressParameters
| asyncSelect | logical OR of OodleXAsyncSelect flags determine how the async is run
|
| packedDataPtr | pointer to LZ compressed data
|
| packedLen | compressed data length
|
| rawPtr | pointer to memory filled with decompressed data
|
| rawChunkLen | length of decompressed data
|
| checkCRC | if OodleLZ_CheckCRC_Yes, the decompressor checks the crc to ensure data integrity
|
| verbosity | (optional) if not OodleLZ_Verbosity_None, will log some information
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder.
|
| decBufSize | (optional) size of circular buffer starting at decBufBase
|
| pcb | (optional) OodleDecompressCallback called during decompression
|
| pcbData | (optional) user data passed to pcb
|
| writeToFile | (optional) OodleXIOQFile to write raw data to
|
| writeToFileStartPos | (optional) file position where writeToFile should start (must be OODLEX_IO_MAX_ALIGNMENT aligned)
|
| writeHandleGroup | (optional) OodleAsyncGroup handle which the write handle is put into
|
| writeHandleGroupIndex | (optional) index in writeHandleGroup to use; must previously be OODLEX_ASYNC_HANDLE_PENDING
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Discussion
:return OodleXHandle to the operation, or OodleXHandle_Null for invalid arguments
Start an async LZ decompress with the runner specified in asyncSelect.
A Narrow decompress means the entire decompression is done on one thread.
Data will always be decompressed sequentially, eg. in order.
rawChunkLen can be less than the entire original block, if it is a multiple of OODLELZ_BLOCK_LEN.
If provided, the OodleDecompressCallback is called as quanta of raw data are available. The callback
may be called more often than OODLELZ_BLOCK_LEN granularity.
rawPtr and packedDataPtr memory blocks passed to this function must be kept alive for the duration of the async.
NOTE !! : if writeToFile is provided, the writes are async and are NOT necessarily done when
the returned handle is done; the returned handle is for the decompress. The handle for the write can
be retrieved by passing in writeHandleGroup. You must not free the buffer being written until the
write operation is done.
| Oodle2 Core vs Oodle2 Ext |
|
|
Oodle Data Compression comes in two libraries on most platforms : Core and Ext.
(On some embedded and mobile platforms, Ext is not provided)
(note Oodle Network is in its own SDK and libraries; see Oodle Network and Data SDK separation)
Oodle2 Core lib provides simple memory->memory synchronous operations.
(eg. OodleLZ_Compress)
Oodle2 Ext extends the Core library by adding asynchronous and multi-threaded compression,
as well as file IO and job scheduling.
(eg. OodleXLZ_ReadAndDecompress_Stream_Async)
Core lib APIs start with Oodle_ , Ext lib APIs start with OodleX_
In general, it's recommended that you use OodleX in your tool chain, and use only Oodle Core
in your shipping runtime.
OodleX provides its own allocator. Oodle Core defaults to using clib for allocations if needed,
or you can replace that with Core plugins.
OodleX must be Init and Shutdown (see OodleX Startup and Shutdown) once in your app. Oodle Core does not
require any initialization or shutdown, you can start calling Core APIs at any time (but not if
you use OodleX! If you use OodleX, then Init must happen before any Oodle call, including any
Core call).
Oodle Core is available on all platforms. Oodle Ext is not available on iOS or Android.
Oodle Core is entirely thread-safe and reentrant with no blocking primitives, because there is no
global shared state.
As much as possible, the Oodle Core functions are "pure". That is, they act only on their arguments
and do only CPU work, they do not interact with the system in any other way. The main exception is
that Core can make calls to logging and allocation through user-provided system plugins (see Core plugins).
It is possible to make all those plugins NULL in your shipping build; see FAQ: How do I use Oodle with no allocator?.
NOTE : on Windows DLL (and other DLL builds) do not import both Ext and Core! The Ext DLL contains
Core as well.
NOTE : Ext automatically plugs itself into Core. You do not need to use the Plugin functions to
install Ext to Core. That is not true of the new split libs Oodle Network and Oodle Texture. If you
want to install OodleX to Net or Texture you have to call the Plugin functions explicitly.
| OodleCore_Plugins_SetAssertion |
|
|
Discussion
Install the callback used by Oodle Core for assertsParameters
Return Value
| return | returns the previous function pointer
|
Discussion
Use this function to install your own display for Oodle Core assertions.
This will only happen in debug builds.
The default implementation in debug builds, if you install nothing, uses the C stderr printf for logging,
except on Microsoft platforms where it uses OutputDebugString.
WARNING : this function is NOT thread safe! It should be done only once and done in a place where the caller can guarantee thread safety.
Discussion
Align down to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x down to OODLEX_IO_MAX_ALIGNMENT
enum OodleLZ_CompressionLevel
{
OodleLZ_CompressionLevel_None = 0,
OodleLZ_CompressionLevel_SuperFast = 1,
OodleLZ_CompressionLevel_VeryFast = 2,
OodleLZ_CompressionLevel_Fast = 3,
OodleLZ_CompressionLevel_Normal = 4,
OodleLZ_CompressionLevel_Optimal1 = 5,
OodleLZ_CompressionLevel_Optimal2 = 6,
OodleLZ_CompressionLevel_Optimal3 = 7,
OodleLZ_CompressionLevel_Optimal4 = 8,
OodleLZ_CompressionLevel_Optimal5 = 9,
OodleLZ_CompressionLevel_HyperFast1 = -1,
OodleLZ_CompressionLevel_HyperFast2 = -2,
OodleLZ_CompressionLevel_HyperFast3 = -3,
OodleLZ_CompressionLevel_HyperFast4 = -4,
OodleLZ_CompressionLevel_HyperFast = OodleLZ_CompressionLevel_HyperFast1,
OodleLZ_CompressionLevel_Optimal = OodleLZ_CompressionLevel_Optimal2,
OodleLZ_CompressionLevel_Max = OodleLZ_CompressionLevel_Optimal5,
OodleLZ_CompressionLevel_Min = OodleLZ_CompressionLevel_HyperFast4,
OodleLZ_CompressionLevel_Force32 = 0x40000000,
OodleLZ_CompressionLevel_Invalid = OodleLZ_CompressionLevel_Force32
};
Discussion
Selection of compression encoder complexityEnumerants
Discussion
Higher numerical value of CompressionLevel = slower compression, but smaller compressed data.
The compressed stream is always decodable with the same decompressors.
CompressionLevel controls the amount of work the encoder does to find the best compressed bit stream.
CompressionLevel does not primary affect decode speed, it trades off encode speed for compressed bit stream quality.
I recommend starting with OodleLZ_CompressionLevel_Normal, then try up or down if you want
faster encoding or smaller output files.
The Optimal levels are good for distribution when you compress rarely and decompress often;
they provide very high compression ratios but are slow to encode. Optimal2 is the recommended level
to start with of the optimal levels.
Optimal4 and 5 are not recommended for common use, they are very slow and provide the maximum compression ratio,
but the gain over Optimal3 is usually small.
The HyperFast levels have negative numeric CompressionLevel values.
They are faster than SuperFast for when you're encoder CPU time constrained or want
something closer to symmetric compression vs. decompression time.
The HyperFast levels are currently only available in Kraken, Mermaid & Selkie.
Higher levels of HyperFast are faster to encode, eg. HyperFast4 is the fastest.
The CompressionLevel does not affect decode speed much. Higher compression level does not mean
slower to decode. To trade off decode speed vs ratio, use spaceSpeedTradeoffBytes in OodleLZ_CompressOptions
| OodleXLZ_Decompress_ThreadPhased_Narrow_Async |
|
|
OodleXHandle OodleXLZ_Decompress_ThreadPhased_Narrow_Async( const void * compBuf,
OO_SINTa compSize,
void * decBuf,
OO_SINTa rawSize,
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OO_S32 circularBufferBlockCount OODEFAULT( - 1 ),
void * scratchBuf OODEFAULT( NULL ),
OO_BOOL synchronous_use_current_thread OODEFAULT( false ) );Discussion
Start an async LZ decompress for ThreadPhase decoding, using 2 threadsParameters
| compBuf | pointer to compressed data
|
| compBufferSize | number of compressed bytes to decode
|
| rawBuf | pointer to output uncompressed data into
|
| rawLen | number of uncompressed bytes to output
|
| checkCRC | (optional) if data could be corrupted and you want to know about it, pass OodleLZ_CheckCRC_Yes
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder. The decBufBase must be a reset point.
|
| decBufSize | (optional) size of circular buffer starting at decBufBase
|
| circularBufferBlockCount | (optional) number of blocks for circular buffer; generally more is faster but takes more memory; < 0 means use default
|
| scratchBuf | (optional) memory to use for scratch; must be OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded() * circularBufferBlockCount ; if NULL will be allocated
|
| synchronous_use_current_thread | (optional) if true, runs on the current thread and uses 1 additional thread; this makes this a synchronous call and won't return until decompression is done (default is to use 2 worker threads and be fully async)
|
Return Value
| return | OodleXHandle to the operation, wait and check status to get result
|
Discussion
Runs a 2-thread Narrow decompress using the Oodle Worker system.
You must wait and delete the return handle, for example with OodleX_WaitAndDelete
This only works on data that has been compressed with a compressor that's eligible for ThreadPhased decode;
check OodleLZ_Compressor_CanDecodeThreadPhased. (currently just Kraken).
See About OodleLZ ThreadPhased Decode
This function (OodleXLZ_Decompress_ThreadPhased_Narrow_Async) does NOT parallelize at seek reset points.
You can however do so yourself externally to calling this function. Simply scan the compressed buffer for
seek points and launch a separate OodleXLZ_Decompress_ThreadPhased_Narrow_Async call on each seek chunk.
ThreadPhased decode is always fuzz safe.
If synchronous_use_current_thread then the returned handle is not async, you may check its status to get the result.
| OODLE_WORKERS_COUNT_ALL_PHYSICAL_CORES |
|
|
Discussion
Make workers for every physical core
eg. in a 6-physical core, 12-hyper-thread system, would make 6 threads
this is usually best for Oodle Data LZ compression work
See also OODLE_WORKERS_COUNT_ALL_HYPER_CORES
| OodleLZ_Compressor_MustDecodeWithoutResets |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor must decode contiguous ranges of buffer with the same Decoder.
That is, most of the compressors can be Reset and restart on any block, not just seek blocks,
as long as the correct window data is provided. That is, if this returns false then the only
state required across a non-reset block is the dictionary of previously decoded data.
But if OodleLZ_Compressor_MustDecodeWithoutResets returns true, then you cannot do that,
because the Decoder object must carry state across blocks (except reset blocks).
This does not apply to seek points - you can always reset and restart decompression at a seek point.
Discussion
Opaque type for an OodleNetwork1UDP_StateDiscussion
OodleNetwork1UDP uses a OodleNetwork1_Shared just like the non-UDP OodleNetwork1
| OodleX_WaitAndDestroyThread |
|
|
Discussion
Wait on thread being complete and free all resourcesDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
| OodleLZ_Decode_ThreadPhase |
|
|
enum OodleLZ_Decode_ThreadPhase
{
OodleLZ_Decode_ThreadPhase1 = 1,
OodleLZ_Decode_ThreadPhase2 = 2,
OodleLZ_Decode_ThreadPhaseAll = 3,
OodleLZ_Decode_Unthreaded = OodleLZ_Decode_ThreadPhaseAll
};
Discussion
ThreadPhase for threaded Oodle decodeDiscussion
Check OodleLZ_Compressor_CanDecodeThreadPhased
(currently only used by Kraken)
See About OodleLZ ThreadPhased Decode
| OodleNetwork1_Shared_SetWindow |
|
|
Discussion
Fill a OodleNetwork1_Shared from provided dataParameters
Discussion
This must be done on both the client and server to fill the OodleNetwork1_Shared object used for compression.
window should be some typical transmitted data. The better you can make it represent the common data seen, the better compression will be. The most common types of packets should be placed at the end of the window.
You must load window from disk somehow, or it could be data that you already have in memory for some other purpose - any data which both the client and server have exactly the same copy of can be used as the compression dictionary. To save and load the window from disk you should generally use one of the standard About OodleLZ compressors.
NOTE : window is not copied ; do not free it! OodleNetwork1_Shared keeps pointers into window, it must be kept allocated while this OodleNetwork1_Shared is in use.
You may call SetWindow multiple times on the same Shared data for purposes of training.
| OodleLZ_Compressor_CanDecodeFuzzSafe |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor is "fuzz safe" which means it can accept corrupted data
and won't crash or overrun any buffers.
Oodle on Android is currently provided as only the Oodle Core lib (no OodleX).
This includes the synchronous LZ compressors, as well as Oodle Network compression.
See Oodle2 Core vs Oodle2 Ext
Oodle on mobile does not include the Optimal level encoders. When OodleLZ_CompressionLevel_Optimal1 or higher
is requested, OodleLZ_CompressionLevel_Normal is used instead.
About Oodle
Oodle is a family of solutions for efficient data compression.
Oodle consists of :
Oodle Data : generic lossless data compression (Kraken, Leviathan, Mermaid, and Selkie)
Oodle Network : packet compression to reduce bandwidth (see About Oodle Network Compression or Getting Started with Oodle Network).
Oodle Texture : encoding for BCN block-compressed GPU textures that dramatically reduces their size.
Oodle Lossless Image : specialized encoder for lossless RGB image encoding. Faster and smaller than PNG.
Oodle Data and Network are both described in this help, but are distributed as separate SDKs.
Oodle Texture and Oodle Lossless Image are distributed separately.
Oodle Data SDK comes as two libraries - Core and Ext. Core provides synchronous memory->memory compressors
that require no initialization and are very portable. Ext provides helpers for async compression and
file IO. Most games should use Oodle2 Core in their runtime and Oodle2 Ext in their tool chain.
(see Oodle2 Core vs Oodle2 Ext)
Oodle Data compresses packages for distribution with state of the art lossless (LZ) data compression.
Oodle provides many different levels of CPU use to compression ratio tradeoffs.
Oodle lets you use the same data compression and packaging formats on all platforms.
(see About OodleLZ or Getting Started with Oodle LZ Data Compression)
Further Reading
See also :
Frequently Asked Questions
Index of Abouts
Introducing the new Oodle Leviathan
Getting Started with Oodle Network
Getting Started with Oodle LZ Data Compression
Oodle2 Core vs Oodle2 Ext
Oodle Network and Data SDK separation
Discussion
Return a const OodleXFileOpsVTable with the base OS implementationsDiscussion
Contains the base file ops functions for the current OS.
Do not change this struct!
| OodleXIOQ_SetInfoByName_Async |
|
|
Discussion
Start an asynchronous SetInfo requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Used to set flags or modtime on a file.
Flags and modTime correspond to OodleXFileInfo:flags and OodleXFileInfo:modTime.
All members of OodleXFileInfo can be set this way, except size; to set size use OodleXIOQ_SetFileSize_Async.
Discussion
void-void callback func pointer
takes void, returns void
enum OodleLZ_CheckCRC
{
OodleLZ_CheckCRC_No = 0,
OodleLZ_CheckCRC_Yes = 1,
OodleLZ_CheckCRC_Force32 = 0x40000000
};
Discussion
Bool enum for the LZ decoder - should it check CRC before decoding or not?Discussion
NOTE : the CRC's in the LZH decompress checks are the CRC's of the compressed bytes. This allows checking the CRc
prior to decompression, so corrupted data cannot be fed to the compressor.
To use OodleLZ_CheckCRC_Yes, the compressed data must have been made with OodleLZ_CompressOptions:sendQuantumCRCs set to true.
If you want a CRC of the raw bytes, there is one optionally stored in the OodleLZ_SeekTable and can be confirmed with
OodleLZ_CheckSeekTableCRCs
enum OodleXFileOpenFlags
{
OodleXFileOpenFlags_Default = 0,
OodleXFileOpenFlags_Buffered = 1,
OodleXFileOpenFlags_NotBuffered = 2,
OodleXFileOpenFlags_WriteCreateDontStomp = 4,
OodleXFileOpenFlags_Force32 = 0x40000000
};
Discussion
OodleXFileOpenFlags specify options when opening filesEnumerants
Discussion
OodleXFileOpenFlags_Default lets Oodle select buffered or unbuffered based on the system and global
settings.
OodleXFileOpenFlags_Buffered files are guaranteed to work with unaligned IO.
OodleXFileOpenFlags_Default and OodleXFileOpenFlags_NotBuffered files require aligned IO on some platforms.
Flags may be combined with logical OR
| OodleXLZ_Compress_AsyncAndWait |
|
|
Discussion
Does OodleXLZ_Compress_Async and OodleXLZ_Compress_Wait_GetResult
| OodleXLZ_Decompress_MakeSeekTable_Wide_Async |
|
|
OodleXHandle OodleXLZ_Decompress_MakeSeekTable_Wide_Async( OO_U32 asyncSelect,
OO_S32 seekChunkLen,
const void * packedDataPtr,
OO_SINTa packedLen,
void * rawArray,
OO_SINTa rawArrayLen,
OodleLZ_FuzzSafe fuzzSafe OODEFAULT( OodleLZ_FuzzSafe_No ),
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OodleLZ_PackedRawOverlap packedRawOverlap OODEFAULT( OodleLZ_PackedRawOverlap_No ),
OodleXIOQFile writeToFile OODEFAULT( 0 ),
OO_S64 writeToFileStartPos OODEFAULT( 0 ),
OodleXHandle * pWriteHandleGroup OODEFAULT( 0 ),
OodleXHandleAutoDelete autoDelete OODEFAULT( OodleXHandleAutoDelete_No ),
const OodleXHandle * dependencies OODEFAULT( NULL ),
OO_S32 numDependencies OODEFAULT( 0 ) );Discussion
Start an async LZ decompress, possibly write raw dataParameters
| asyncSelect | logical OR of OodleXAsyncSelect flags determine how the async is run
|
| seekChunkLen | length of seekChunks used in compression OodleLZ_CompressOptions:seekChunkLen
|
| packedDataPtr | pointer to LZ compressed data
|
| packedLen | compressed data length
|
| rawArray | pointer to memory filled with decompressed data
|
| rawArrayLen | length of decompressed data
|
| checkCRC | if OodleLZ_CheckCRC_Yes, the decompressor checks the crc to ensure data integrity
|
| verbosity | (optional) if not OodleLZ_Verbosity_None, will log some information
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder.
|
| decBufSize | (optional) size of circular buffer starting at decBufBase
|
| packedRawOverlap | (optional) if OodleLZ_PackedRawOverlap_Yes, the compressed data is in the same memory array as the output raw data
|
| writeToFile | (optional) OodleXIOQFile to write raw data to
|
| writeToFileStartPos | (optional) file position where writeToFile should start (must be OODLEX_IO_MAX_ALIGNMENT aligned)
|
| pWriteHandleGroup | (optional) if writeToFile is given, this is filled with an OodleAsyncGroup OodleXHandle containing all the file IO operations
|
| autoDelete | (optional) see OodleXHandleAutoDelete
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | OodleXHandle to the operation, or OodleXHandle_Null for invalid arguments
|
Discussion
Same as OodleXLZ_Decompress_Wide_Async , but makes the seek table for you.
Can be used as a drop-in replacement for OodleLZ_Decompress() but with parallel decoding.
If the data is not parallel-decodable (because it has no seek resets, eg.
OodleLZ_CompressOptions:seekChunkReset was not set) this is slower than just calling OodleLZ_Decompress.
So this should only be used when you believe parallel decoding is possible.
seekChunkLen most follow the rules for Oodle seek chunk lengths. See OodleLZ_MakeSeekChunkLen.
It should be a power of two and greater-equal than OODLELZ_BLOCK_LEN.
Discussion
Compress some data from memory to memory, synchronously, with OodleLZParameters
Return Value
Discussion
Performs synchronous memory to memory LZ compression.
In tools and when compressing large inputs in one call, consider using
OodleXLZ_Compress_AsyncAndWait (in the Oodle2 Ext lib) instead to get parallelism. Alternatively,
chop the data into small fixed size chunks (we recommend at least 256KiB, i.e. 262144 bytes) and
call compress on each of them, which decreases compression ratio but makes for trivial parallel
compression and decompression.
You can compress a large buffer in several calls by setting dictionaryBase to the start
of the buffer, and then making rawBuf and rawLen select portions of that buffer. As long
as rawLen is a multiple of OODLELZ_BLOCK_LEN, the compressed chunks can simply be
concatenated together.
The buffer that compBuf points to must have a certain minimum size that is returned by
OodleLZ_GetCompressedBufferSizeNeeded. This size is always more than rawLen, usually by a few bytes
for every 256kb of input data. The "compressed" data can end up slightly larger than the input due to
internal stream headers.
If scratchMem is provided, it will be used for the compressor's scratch memory needs before OodleMalloc is
called. If the scratch is big enough, no malloc will be done. If the scratch is not big enough, the compress
will not fail, instead OodleMalloc will be used. OodleMalloc should not return null. There is currently no way
to make compress fail cleanly due to using too much memory, it must either succeed or abort the process.
If scratchSize is at least OodleLZ_GetCompressScratchMemBound , additional allocations will not be needed.
See About OodleLZ for tips on setting the compression options.
If dictionaryBase is provided, the backup distance from rawBuf must be a multiple of OODLELZ_BLOCK_LEN
If OodleLZ_CompressOptions:seekChunkReset is enabled, and dictionaryBase is not NULL or rawBuf , then the
seek chunk boundaries are relative to dictionaryBase, not to rawBuf.
| OodleNetwork1TCP_State_Size |
|
|
Discussion
Returns the size of memory required for an OodleNetwork1TCP_State objectDiscussion
Shared and State are allocated with malloc( Size() )
Discussion
Align down to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x down to OODLEX_IO_MAX_ALIGNMENT
| OodleXIOQ_CopyFile_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_CopyFile_Async
| OodleLZ_CompressOptions_Validate |
|
|
Discussion
Clamps the values in pOptions to be in valid range
OodleXMalloc memory allocators.
About OodleLZ Hydra
Oodle Hydra is a meta-compressor which automatically selects Leviathan, Kraken, Mermaid, or Selkie on a
per-block basis.
When you decode a file that was compressed with Hydra, it will decode as one or several of those other
compressors.
Hydra makes its decision by scoring each compressor for its space-speed performance. What this means is
Hydra automatically makes good decisions about using the slower compressors only when they are worth it.
That is, they must provide a good return in terms of bytes saved in exchange for the increase in decode
time. Exactly what qualifies as "worth it" is determined by you, via the OodleLZ_CompressOptions:spaceSpeedTradeoffBytes
parameter.
With spaceSpeedTradeoffBytes around 256 , Hydra is roughly comparable to Kraken. As you dial it lower,
Hydra will give more compression but slower decodes. Around 1500 is comparable to Mermaid, and around 50
is comparable to Leviathan. In between you can hit a balance that's somewhere between those compressors.
Hydra takes more time to encode because it is considering many compressors.
Hydra always beats every other Oodle compressor, it provides the best of them.
| FAQ: How does OodleLZ compare with other compressors ? |
|
|
In general, Oodle LZ is designed to get compression as good as anything in the world, while decoding much MUCH faster.
Usually 2-10X faster!
Kraken is the compressor you should try first. It offers excellent compression ratio and very high decode speeds.
It's a great compromise for game data loading.
Leviathan gets a bit more compression than Kraken (similar to 7zip/LZMA, or Oodle LZNA), but is a bit slower to decode than Kraken.
It's way faster than anything else with similar compression. Leviathan can be a bit slow to encode.
Mermaid & Selkie are some of the fastest decompressors in the world. They provide less compression than Kraken but
super fast decodes.
Mermaid & Selkie (at the fast OodleLZ_CompressionLevel settings) are also very fast to encode.
Here's a chart showing the compression ratio of various files of different types :
And a chart of the decode speed :
(all compressors were run in max compress mode; per-file hand tweakable of options was not done for any of the compressors;
multimedia and x86 filters were disabled for all compressors)
The best way for you to evaluate the Oodle compressors is just to run them on your own data and see what you get.
Try example_lz_chart : Example that makes a chart of OodleLZ options (in the bin dir) for an easy way to get a report of Oodle's performance.
| OodleLZ_GetCompressScratchMemBoundEx |
|
|
Discussion
Return an estimate for the amount of scratch mem used by OodleLZ_CompressParameters
Discussion
Returns either a worst-case or typical scratch memory estimate for the given compressor, options
and input size.
When a worst-case scratch memory estimate exists, passing that much scratch memory to
OodleLZ_Compress is guaranteed to not do any allocations, provided the parameters match those
of the compression call.
"Typical" memory bounds are not hard guarantees but will usually not result in any extra allocations.
For rawLen pass at least the maximum size you will ever encode. If your data is divided into chunks,
pass the chunk size. If you will encode full buffers of unbounded size, pass -1.
Some options and levels may not have simple finite bounds. Then OODLELZ_SCRATCH_MEM_NO_BOUND is returned
and the call to OodleLZ_Compress may use the allocator even if infinite scratch memory is provided.
Currently this applies to all the Optimal levels.
When OODLELZ_SCRATCH_MEM_NO_BOUND is returned, you can still pass in scratch mem which will be used before
going to the plugin allocator.
| OodleX Startup and Shutdown |
|
|
- Defines
- Enumerants
- Structures
- Functions
| Tips for benchmarking a compressor |
|
|
You're about to evaluate Oodle (thanks for having a look!) or some other compressor.
Before you start, consider these tips :
- Time only the compressor.
Place your time measurements only around the compressor. Not IO, not your parsing, not mallocs, just the compress or
decompress calls. I understand that in the end what you care about is total time to load, but there can be a lot of
issues there that need fixing, and they can cloud the comparison of just the compression part. eg. if your parsing is
really slow, that will dominate the CPU time and hide the differences between the compressors.
- Time what you actually care about.
If you care about decode time, time the decompression. If you care about encode time, time compression. If you care about
round-trip time, add the two times. Compressors are not just "fast" or "slow" at both ends, you can't time encoding and
decide that it's a fast or slow compressor if what you care about is decoding.
- Choose the right options.
Most compressors have the ability to target slightly different use cases. The most common option is the ability to trade off
encode time vs. compression ratio. So, if what you care about is smallest size, then run the compressor at its highest encode
effort level. It can be tricky to get the options right in most compression libraries; we are woefully non-standardized and
not well documented. Aside from the simple "level" parameter, there may be other options that are relevant to your goals,
perhaps trading off decompressor memory usage, or decompression speed.
With Oodle the best option is always to email us and ask what options will best suit your goals.
- Run apples-to-apples (threads-to-threads) comparisons.
It can be tricky to compare compressors fairly. As much as possible they should be run in the same way, and they should
be run in the way that you will actually use them in your final application. Don't profile them with threads if you will
not use them threaded in your shipping application.
Threads are a common problem. Compressors should either be tested all threaded (if you will use threads in your final application),
or all non-threaded. Unfortunately the defaults are not the same. "lzma" (7z) and LZHAM create threads by default. You have
to change their options to tell them to not create threads. The normal Oodle_Compress calls will not use threads by default,
you have to specifically call one of the Async threaded routines. So either set everything to not use threads, or set everything to
use threads, so the comparison is on equal footing. (my personal preference is to benchmark everything without threads to compare
single-threaded performance, and you can always add threads for production use)
- Take the MIN of N run times.
To get reliable timing, you need to run the loop many times, and take the MIN of all times. The min will
give you the time it takes when the OS isn't interrupting you with task switches, the CPU isn't clocking-down
for speedstep, etc. I usually do 30 per core but you can probably get a way with a bit less.
On some modern cores that do short term boost, or on ARM big/little cores, it can be better to take the MEDIAN
(not the average) of many runs, or to exclude the top 10% fastest runs then take the min.
If possible, use BIOS and OS settings to put the procesor in a more stable mode.
Assuming you are now doing N loops, you need to invalidate the cache between iterations. If you don't,
you will be running the compressor in a "hot cache" scenario, with some buffers already in cache.
Most likely the real world performance you actually care about is the cold cache performance, so
invalidate the cache in the timing loop.
This has the biggest effect on small buffers. It's fine to do your timing in either hot cache or cold cache,
the important thing is consistency - don't care numbers from a hot cache run vs a cold cache run, particularly on
small buffers.
- Don't pack a bunch of files together in a tar if that's not how you load.
It may seem like a good way to test to grab your bunch of test files and pack them together in a tar (or zip -0 or similar package)
and run the compression tests on that tar. That's a fine option if that's really how you load data in your final application - as
one big contiguous chunk that must be loaded in one big blob. But most people don't. You need to test the compressors in the same
way they will be used in the final application. If you load whole file at a time, test the compressors on whole file units. Many
people do loading on some kind of paging unit, like perhaps 1 MB chunks. If you do that, then test the compressor on the same thing.
- Choose your test set to proportionally represent your real data load.
If you could test on the entire set of buffers that your final application will load, that would be an
accurate test. (though actually, even that is a bit subtle, since some buffers are more latency sensitive
than others, so for example you might care more about the first few things you load to get into a running
application as quickly as possible). That's probably not practical, so you want to choose a set that is
representative of what you will actually load. Don't exclude things like already-compressed files (JPEGs
and so on) if you will be running them through the compressor. (though consider not running them your
compressed-file loading path, in which case you should exclude them from testing). It's pretty hard to
get an accurate representative sample, so it's generally best to just get a variety of files and look at
individual per-file results.
A common mistake is to test on standard corpora like Silesia or Canterbury that have a majority of their
data in ASCII text. You probably do not the majority of your bytes spent on text, so those are very poor
reflections or real world data loads.
- Look at the spectrum of results, not the sum.
After you run on your test set, don't just add up the compressed sizes and times to make a "total" result. Sums can be misleading.
One issue is there are some large incompressible files, they can hide the differences on the more compressible files. But a bigger and
more subtle trap is the way that sums weight the combination of results. A sum is a weighting by the size of each file in the test
set. That's fine if your test set is all of your data, or is a perfectly proportionally representative sampling of all of your data
(a subset which acts like the whole). But most likely it's not. It's best to keep the results per file separate and just have a look
at individual cases to see what's going on, how the results differ, and try not to simplify to just looking at the sum.
- If you do sum, sum time not speed, sum size not ratio.
Speed (like mb/s) and ratio (raw size/comp size) are inverted measures and shouldn't be summed. What you
actually care about is total compressed size, and total time to decode. So if you run over a set of files,
don't look at "average speed" or "average ratio" , because those are inverted meaures that will oddly weight
the accumulation. Instead accumulate total time to decode, total raw size, and total compressed size, and
then if you like you can make "overall speed" and "overall ratio" from those total.
- Try not to malloc in the timing loop.
Your malloc might be fast, it might be slow, it's best to not have that as a variable in the timing. In
general try to allocate the memory for the compressor or decompressor outside of the timing loop.
(In Oodle this is done by passing in your own pointer for the "decoderMemory" argument of OodleLZ_Decompress).
That would be an unfair test if you didn't also do that in the final application - so do it in the final application too!
(similarly, make sure there's no logging inside the timing loop).
- Consider excluding almost-incompressible files.
This is something you should consider for final shipping application, and if you do it in your shipping
application, then you should do it for the benchmark too. The most common case is already-compressed files
like JPEG images and MP3 audio. These files can usually be compressed slightly, maybe saving 1% of their
size, but the time to decode them is not worth it overall - you can get more total size savings by running a
more powerful compressor on other files. So it's most efficient to just send them uncompressed.
- Tiny files should either be excluded or packed together.
There's almost never a use case where you really want to compress tiny files (< 16k bytes or so) as
independent units. There's too much per-unit overhead in the compressor, and more importantly there's
too much per-unit overhead in IO - you don't want to eat a disk seek to just to get one tiny file. So
in a real application tiny files should always be grouped into paging units that are 256k or more, a size
where loading them won't just be a total waste of disk seek time. So, when benchmarking compressors you
also shouldn't run them on tiny independent files, because you will never do that in a shipping application.
- Beware of trying to time IO.
It is attractive to try to time the entire loading process (IO + decompression + processing) to see how
Oodle affects your total load time. While this is a nice idea, it is fraught with peril in practice.
IO timing is notoriously inconsistent, and the times you measure may not reflect real user times at all.
(for example, if you measuring timing repeatedly, you may be just loading data from the system disk
cache; on a console you may be timing through a simulator or host FS which doesn't reflect real speeds;
etc.) Because of this I recommend timing just the compression portion of the loading operation, or at
least timing both aspects. To really time IO right you need to make many runs on different hardware,
after fresh reboots, it's quite a lot of work.
- Compare to the provided Oodle executables
Oodle provides example_lz_chart : Example that makes a chart of OodleLZ options and ozip (About Oodle ozip) as pre-built executables (in the bin dir)
that can be used for benchmarking. Test your code to make sure your results agree with them.
They are also available as source code so you can see how the benchmarking is done.
- One good number is better than a million bad ones.
Before you go and try to be all "big data" and run over a giant corpus, make sure you can benchmark just
one file and get correct results. Compare it to the published Oodle benchmark results, which are very
carefully made, and if it doesn't agree make sure you know why. Oodle is run on some public domain data
sets, such as the Silesia corpus, which you can get here :
http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
if your numbers on Silesia don't agree with the standard Oodle numbers, look into why.
| OodleXIOQ_OpenWriteWholeFileClose_Async |
|
|
Discussion
Start a high level IO request to open a file, write a buffer, and close it.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
Performs OodleXIOQ_OpenForWriteCreate_Async, OodleXIOQ_Write_Async, and OodleXIOQ_CloseFile_Async.
You might also want to enqueue a OodleXIOQ_FreeBufferIOAligned_Async after this, but it is not done for you.
See also FAQ: I write a file with IOQ but the contents are garbage?.
The OodleXHandle returned is not done until the entire compound operation is done.
Oodle API's to help with debugging and diagnostics.
- Defines
- Enumerants
- Functions
- Typedefs
| How to build and use the Oodle examples |
|
|
Oodle ships with several examples.
Each file in the "examples" directory that starts with "example_" is an
independent example which can be built on its own.
The Oodle examples are all console applications. So on Windows with MSVC you should
create an empty Win32 console app project to get started.
To compile the examples, set your include path so that Oodle sdk include
directory is in the path (so that "oodle.h" can be found).
To link the examples, you must specify one of the oodle libs. See About Oodle on Platforms
for the name of the Oodle lib on your platform.
To run the examples on Windows, they must be able to find the Oodle dll (distributed
in the "redist" directory). You can either run the examples in the redist directory,
or copy the Oodle dll do the directory you wish to run the examples from. It is not
advised to copy the Oodle dll into the windows system directory.
The Oodle examples write log files and threadprofile logs to a directory called "c:\oodlelogs" ;
this directory will be created if it does not exist. If you want them to write to another
place (or not write any logs) you can change the example code.
The examples use some default file names to find some data to work on; the content of these files is not important
and they are not distributed with Oodle. You can copy in any file of your choice to serve as
test data for the examples. Some of the examples load a file named "oodle_example_input_file" ; if this
file is not found, they will create one. You can also put your own test file in its place.
Try starting with example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core
Discussion
scan compressed LZ stream to fill the seek tableParameters
Return Value
Discussion
pTable must be able to hold at least OodleLZ_GetSeekTableMemorySizeNeeded
seekChunkLen must be a multiple of OODLELZ_BLOCK_LEN.
seekChunkLen must match what was in CompressOptions when the buffer was made, or any integer multiple thereof.
| About Compression Scratch Memory |
|
|
About Compression Scratch Memory
OodleLZ_Compress takes a "scratch memory" argument that lets the user provided a memory buffer that the compressors
can use instead of allocating their own using allocation callback.
This has long been used at the lower compression levels that have fairly predictable memory usage.
Starting with Oodle 2.9.9, Optimal1 and higher levels (which use more complicated data structures that have less
predictable mmeory usage) use scratch memory much more extensively as well, which can greatly reduce the number of
large allocations performed by the encoder.
In the common scenario where Oodle Data is used to compress individual chunks of <=256KB, allocations can often be
avoided completely provided a large enough buffer is provided. For <=256KB chunks, 32MB is usually more than sufficient
for all codecs and all compression levels. (Larger inputs or threaded/parallel encoding use cases are more complicated
and will, in general, still do allocations.) If the scratch memory is insufficient, Oodle will use regular allocations
and frees once that memory area is exhausted.
The main advantage of providing scratch memory instead of letting the encoder do its own allocations is that this allows
a single large allocation to be shared over many consecutive encodes, reduction memory allocation/freeing overhead
significantly. Optimal1 and higher tend to allocate a few big data structures, and these large allocations tend to have
fairly high overhead otherwise.
For the intended use case (encoding lots of small independent chunks), we've seen encoding speed-ups of around 10%
from using scratch memory when using a single thread. With many threads, the impact can be massive. On 64-core machines
running Windows (as of Jan 2023), we've seen speed-ups of over 8x from providing scratch memory instead of using the
default Windows heap, because otherwise large allocations get passed through to the kernel which then eventually serializes
on page table updates when the newly allocated memory is first accessed.
OO_BOOL Oodle_CheckVersion( OO_U32 oodle_header_version,
OO_U32 * pOodleLibVersion OODEFAULT( NULL ) );Discussion
Check the Oodle lib version against the header you are compiling withParameters
Return Value
Discussion
If you use the Oodle2 Ext lib,, OodleX_Init does it for you. But if you want to check that you have a
compatible lib before trying to Init, then use this.
Discussion
Create a OodleLZDecoderParameters
| compressor | the type of data you will decode; use OodleLZ_Compressor_Invalid if unknown
|
| rawLen | total raw bytes of the decode (or <= 0 for any/unknown, but Reset for each buffer)
|
| memory | (optional) provide memory for the OodleLZDecoder object (not the window)
|
| memorySize | (optional) if memory is provided, this is its size in bytes
|
Return Value
Discussion
If memory is provided, it must be of size OodleLZDecoder_MemorySizeNeeded. If it is NULL it will be
allocated with the malloc specified by Core plugins.
Free with OodleLZDecoder_Destroy. You should Destroy even if you passed in the memory.
Providing compressor lets the OodleLZDecoder be the minimum size needed for that type of data.
If you pass OodleLZ_Compressor_Invalid, then any type of data may be decoded, and the Decoder is allocated
large enought to handle any of them.
If you are going to pass rawLen to OodleLZDecoder_Reset , then you can pass 0 to rawLen here. To make a Decoder
to use with many buffer sizes, pass either <= 0 (for infinite) or the largest buffer size you can see. Then call
Reset() with the correct buffer size before starting to decode each buffer.
See OodleLZDecoder_DecodeSome for more.
| Oodle2 Core API Documentation |
|
|
| OODLEX_FILEINFO_MODTIME_INVALID |
|
|
Discussion
Invalid value for OodleXFileInfo:modTime
| OODLELZ_SCRATCH_MEM_NO_BOUND |
|
|
Discussion
Scratch mem size when bound is unknown.
Installed allocator may be used no matter how much scratch mem you provide.
| OodleNetwork1_SelectDictionaryFromPackets_Trials |
|
|
OO_BOOL OodleNetwork1_SelectDictionaryFromPackets_Trials( void * dictionary_to_fill,
OO_S32 dictionary_size,
OO_S32 htbits,
const void * * dictionary_packet_pointers,
const OO_S32 * dictionary_packet_sizes,
OO_S32 num_dictionary_packets,
const void * * test_packet_pointers,
const OO_S32 * test_packet_sizes,
OO_S32 num_test_packets,
OO_S32 num_trials,
double randomness_percent,
OO_S32 num_generations );Discussion
Multi-Trial variant of OodleNetwork1_SelectDictionaryFromPacketsParameters
| num_trials | number of trials per generation; 5-20 is a good starting range
|
| randomness_percent | randomness of trials; this is a percent of standard, 100 is a good default; 50-200 is a useful range
|
| num_generations | number of generations; 1 is fine, more is slower
|
Return Value
| return | true on success, false otherwise.
|
Discussion
This function runs the packet selector of OodleNetwork1_SelectDictionaryFromPackets
repeatedly with some randomness. Variation of packet selection can sometimes give
slightly better dictionaries.
The success of the trial is measured on the "test_packet" set. Make sure that the
"dictionary_packet" set are independently randomly drawn from the packet source data
wrst the "test_packet" set so that over-training degeneracies are not created.
On platforms where OodleNetwork1_SelectDictionarySupported returns false, this
function is not supported.
It's advised to run this function in 64-bit, as it can use a lot of memory.
#define OodleXLog_Printf OodleXLog_Printf_Raw(verboseLevel,__FILE__,__LINE__,##__VA_ARGS__)
Discussion
OodleXLog_Printf lets you write to Oodle's log.Discussion
Use like printf : OodleXLog_Printf(verbose,fmt,arg1,arg2,...)
What kind of output is produced from this depends on the bit flags set in OodleXLog_SetState.
If the global verbose level set by OodleXLog_SetVerboseLevel is < verboseLevel passed here,
the message is supressed.
OodleXLog_Printf_vN(fmt,...) is the same as OodleXLog_Printf(N,fmt,...)
| OODLEX_ASYNC_HANDLE_ERROR |
|
|
Discussion
OodleXHandle to a special always-error handle.
Calls to OodleX_GetStatus on this handle value will return &OodleXStatus_Error.
This handle must not be deleted! Do not call OodleX_Wait on it with deleteIfDone = true.
| OodleXIOQ_ReadMallocWholeFile_GetResult |
|
|
Discussion
Finish a OodleXIOQ_ReadMallocWholeFile_Async requestParameters
| req | the OodleXHandle to the OodleXIOQ_ReadMallocWholeFile_Async request
|
| andDeleteIfDone | if true and the returned status is >= Done the handle will be deleted
|
| pPtr | filled out with the buffer allocated by OodleXIOQ_ReadMallocWholeFile_Async
|
| pSize | (optional) filled with the file size
|
Return Value
| return | the status ; if <= OodleXStatus_Pending , result pointers are set to null
|
Discussion
OodleXIOQ_ReadMallocWholeFile_GetResult does NOT wait on the handle.
See OodleXIOQ_ReadMallocWholeFile_Async
Oodle2 for Mac is provided as a shared library. (change in 2.5.1 : static libraries now provided as well)
lib/liboo2coremac.version.dylib
lib/liboo2coremac64.version.dylib
lib/liboo2extmac.version.dylib
lib/liboo2extmac64.version.dylib
Where "version" is the current Oodle version (2.9.11).
Oodle for Mac is provided in 32 and 64 bit builds.
The debug build of the Oodle lib is also provided. Generally the release build of Oodle should be linked with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems.
OodleX for Mac tries to write a log file to "/var/log/oodle/" by default. If you don't want this, you
may disable logging or change the log location in OodleXInitOptions. On most systems, Oodle will fail to
create the "oodle" subdir in "var/log/" due to lack of permissions; you must create that dir for Oodle.
When Oodle fails to write the log to the desired location, it will instead write it to "." (current directory).
| OODLE_WORKERS_COUNT_ALL_HYPER_CORES |
|
|
Discussion
Make workers for every hyper-thread
eg. in a 6-physical core, 12-hyper-thread system, would make 12 threads
this is usually best for Oodle Texture work
See also OODLE_WORKERS_COUNT_ALL_PHYSICAL_CORES
| OodleXIOQ_GetErrorDetails |
|
|
Discussion
Convert an OS error code into a text messageParameters
Return Value
| return | bool for success/failure
|
Discussion
fills out pMessage with a text description of the error (if available).
Discussion
Job function pointer for Plugin Jobify systemDiscussion
takes void pointer returns void
Discussion
Opaque weak reference to Oodle asynchronous objects Discussion
Any op which returns an OodleXHandle can be used in OodleX_Wait or as a dependency for other ops.
See OodleX async handle operations
| FAQ: How do I use Oodle with no allocator? |
|
|
If you want to use Oodle2 Core without it doing any allocations itself, you may still do
decompression.
1. Don't use OodleX. Use Oodle Core only. OodleX requires allocators and installs its
own.
2. The default allocator in Oodle Core is clib. If you want to ensure the clib allocators
are never used, you may call OodleCore_Plugins_SetAllocators with NULL function pointers to
ensure all allocations are forbidden. Note that if you call anything in Oodle that
attempts to allocate, it will be a fatal error. For production it's safer to just leave
the clib allocator in place; try setting the allocators to NULL only in a developer's debug
build to verify that Oodle is not calling the allocator. (See example_lz_noallocs : Example demonstrating Oodle compression with no allocations)
3. To do LZ decoding, call OodleLZDecoder_Create or OodleLZ_Decompress, and provide pre-allocated
memory for the decoder with the decoderMemory argument, of size OodleLZDecoder_MemorySizeNeeded.
4. To do LZ encoding, call OodleLZ_Compress with scratchMem large enough for all the encoder's
memory needs. Call OodleLZ_GetCompressScratchMemBound to find the size needed.
If the scratchMem you provide is not big enough, the encoder will use the installed
allocator function (and stop the process if you made it null). Note that only the new LZ's and
levels equal to or below "Normal" can be made allocation free, the old compressors and Optimal
levels cannot.
5. You can also do OodleNetwork1 network packet encoding and decoding from previously
trained states without Oodle doing any internal allocations (you provide all required
memory). You cannot do the off-line training or dictionary selection, those require an allocator.
6. The encoder & decoder only use the provided memory as scratch space, not as retained data storage.
This means you can reuse it between calls, it can be allocated only once and used as scratch each time.
Note however that each thread must have its own scratch memory, it cannot be used simultaneously by
multiple threads.
7. See example_lz_noallocs : Example demonstrating Oodle compression with no allocations
| OodleLZ_Compressor_RespectsDictionarySize |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor obeys OodleLZ_CompressOptions:dictionarySize which limits
match references to a finite bound. (eg. for sliding window decompression).
All the new codecs do (Kraken,Mermaid,Selkie,Leviathan). Some old codecs don't.
Discussion
Start a read requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Starts an async read into memory at file offset _position , of _size bytes.
To do unaligned reads, use OodleXIOQ_ReadUnalignedAdjustPointer_Async , or
simply read a larger amount, and use OodleX_IOAlignDownS64 on position and OodleX_IOAlignUpS64 on _size.
The read is not done when OodleXIOQ_Read_Async returns. You must not free memory until the read is done, as reported by
the handle returned;
| OodleXIOQ_OpenWriteWholeFileCloseTempName_Async |
|
|
Discussion
Start a high level IO request to open a file, write a buffer, close it, and rename it.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
This is the same as OodleXIOQ_OpenWriteWholeFileClose_Async , but the writing is done to a temp file, and then
renamed to name at the end, like OodleXIOQ_CloseFileRename_Async. The rename is only done if the writing succeeded.
Discussion
Align up to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x up to OODLEX_IO_MAX_ALIGNMENT
| OodleFileNotFoundIsAnError |
|
|
enum OodleFileNotFoundIsAnError
{
OodleFileNotFoundIsAnError_No = 0,
OodleFileNotFoundIsAnError_Yes = 1,
OodleFileNotFoundIsAnError_Force32 = 0x40000000
};
Discussion
Bool for whether a file not found is a completion status of OodleXStatus_Error or OodleXStatus_Done
Discussion
Clear any errors on the fileParameters
| file | the IOQFile to query
|
Discussion
Wipe out any previous errors recorded on the file, so that OodleXIOQ_GetLastError now returns zero.
| About Oodle Job Threading Plugins |
|
|
About Oodle Job Threading Plugins
Oodle Core can run multi-threaded with a user-supplied job system via OodleCore_Plugins_SetJobSystem
Oodle Core does not provide its own threading (or IO). It is as low level and minimal as possible, doing primary
memory to memory compression and decompression. High level and system-dependent functionality is kept in OodleX.
If you use OodleX, it plugs in its own job system to Oodle Core automatically in OodleX_Init.
If you do not use OodleX, you may plug in your own job system to get threading of Oodle Core.
Some examples are provided of common job systems (see example_jobify_gcd.mm , example_jobify_linuxtbb.cpp, or example_jobify_win32tp.cpp).
Currently Oodle Core uses the Job plugin system to parallelize Optimal level encoding in OodleLZ_Compress.
It is controlled via the OodleLZ_CompressOptions:jobify option (see OodleLZ_Jobify).
The default setting of OodleLZ_CompressOptions:jobify, OodleLZ_Jobify_Default, is to use a job system for
multithreading if one is provided.
When a job system is provided, encoding of buffers larger than one OODLELZ_BLOCK_LEN (256 KB),
at OodleLZ_CompressionLevel greater or equal to OodleLZ_CompressionLevel_Optimal1, will use
multiple threads.
If you are encoding small chunks (less or equal to OODLELZ_BLOCK_LEN), or if you are already encoding lots of files
at the same time with multiple threads, or if you are encoding below level Optimal1, Jobify will not do much for you.
If you are encoding only one file at a time (or care about latency rather than throughput), if you use level >= Optimal1,
and you encode large files, Jobify will improve encode speed.
Jobify summary :
- What is currently threaded? Only encoding (OodleLZ_Compress), of the new LZ codecs (Kraken,Mermaid,Selkie,Leviathan), at level Optimal1+ , only with more than 1 BLOCK
- Jobify does nothing if you encode single BLOCKS or smaller (256 KB)
- if you parallel encode many files at a time (which you should), further encoder threading with Jobify
will be a big benefit (but also doesn't hurt)
- Jobify uses Wait() on workers from the calling thread.
if your calling thread is already a worker, you must be able to wait-from-worker
which many thread systems cannot do
- Jobify default (OodleLZ_Jobify_Default) is "yes do jobify if a thread system is provided"
so you do not need to set any options to get threaded encoding enabled. If you want to turn off threaded
encoding, even though you provided a thread system, set OodleLZ_CompressOptions:jobify to OodleLZ_Jobify_Disable.
- OodleX provides its thread system automatically, so if you use OodleX then jobify will just be on by default without you doing anything
- example plugin job system implementations are provided
(see example_jobify_gcd.mm , example_jobify_linuxtbb.cpp, or example_jobify_win32tp.cpp).
- Note that when possible you should paralellelize at the chunk level or the file level. This is provides
better parallelism than Jobify.
If you provide your own plugins for Jobify, be aware of the requirements on RunJob and WaitJob :
- RunJob returns an async handle that can be passed to WaitJob. That handle should be unique until it is returned by
the paired call to WaitJob.
- RunJob may be passed dependencies. It should not start its job function until those depenencies are done, but this is
not the same as WaitJob - should not delete those dependency jobs.
- WaitJob may be called before or after the job is done. It should wait on the job (if it is still running), then
clean it up and release the job handle.
- Job handle value 0 is reserved to mean "none"; if RunJob returns 0 it means the work was done synchronously.
- WaitJob will only be called from the original thread that called OodleLZ_Compress. It will not be called from a job
function we invoked.
Some thread systems cannot wait from a worker thread. If you are running OodleLZ_Compress from a worker thread, be aware
then WaitJob may be called on that thread. If Jobify uses the same thread system, WaitJob can be a deadlock.
One option is to use a separate thread pool for your top level parallelism (parallelizing on files or chunks), and a
separate pool for Jobify. That way thread pool 1 only waits on thread pool 2. There is no wait-on-workers from within the
same pool.
OodleX worker thread system supports wait-from-worker so it is okay to use Jobify and OodleLZ parallelism
For example, if you call OodleXLZ_Compress_Async with OodleLZ_CompressOptions:seekChunkReset, it will
parallellize at the seek chunk level, and then again at the jobify level (if each seek chunk is bigger than 1 block).
This means the jobify-level parallelism will call WaitJob on the worker threads that were spawned for seek-level
parallelism. This may not work in some thread systems but is okay with OodleX.
(note that OodleX supports wait-from-worker only for Jobs; you should not OodleX_Wait on non-Job handles from inside a Job,
but you can use them as dependencies)
Discussion
Get the current echo FILEReturn Value
Discussion
See OodleXLog_SetEcho
Discussion
Get the last error on a fileParameters
| file | the IOQFile to query
|
Return Value
| return | the last error on the file (0 for none)
|
Discussion
IO operation errors are tracked on the file to simplify error tracking.
Individual operation errors can be queried with OodleXIOQ_GetStatus.
The error code returned can be processed with OodleXIOQ_GetErrorEnum or OodleXIOQ_GetErrorDetails.
| Getting Started with Oodle LZ Data Compression |
|
|
Oodle compresses generic data with lossless (LZ) data compression.
Oodle provides many different levels of speed vs compression ratio tradeoffs.
Oodle lets you use the same data compression and packaging formats on all platforms.
(see About OodleLZ for more)
If you're benchmarking Oodle compression, please consider Tips for benchmarking a compressor
To sample a range of Oodle compressor speeds & compression ratios, try example_lz_chart : Example that makes a chart of OodleLZ options.
example_lz_chart is now provided as a pre-built executable on desktop platforms. Just go run it
on one of your data files to get a sample of Oodle's performance.
The compressors offer a variety of tradeoffs to suit different needs (see About OodleLZ). The
higher CompressionLevel always provides smaller file sizes at the expense of longer encode times.
You can also use the "ozip" command line utility to test Oodle without building any code.
ozip lets you try various Oodle options and test compression ratios and performance with the ozip -b benchmark mode.
See About Oodle ozip.
Feel free to contact oodle@radgametools.com for help on deciding which compressor is best for your
needs.
Try building an example. example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core is a good place to start. It includes only "oodle2.h"
from the include dir. Link it with the appropriate oo2core lib from the "lib" directory.
(on Windows, you also need the DLL from the "redist" directory).
Now run it on one of your game data files to try it.
example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core uses the compressor OodleLZ_Compressor_Kraken , and the encode effort
OodleLZ_CompressionLevel_Normal. Kraken generally provides a good tradeoff between compression ratio
and decode speed. A higher CompressionLevel trades off encode speed for greater compression ratio.
Using Oodle for simple memory to memory compression
To compress memory->memory synchronously, you call OodleLZ_Compress. To decompress the same way you call
OodleLZ_Decompress. These are in the API group OodleAPI_LZ_Compressors in Oodle Core.
To decode parts of the stream into a buffer incrementally use OodleLZDecoder_DecodeSome
Let's walk through a simple memory to memory encode and decode with Oodle's LZ compressors.
The LZ compressors are lossless and compress any kind of data.
For this simple synchronous operation, we can use just Oodle Core, not OodleX. No intialization is required.
We can simply make the call to do memory->memory compression :
SINTa comp_len = OodleLZ_Compress(OodleLZ_Compressor_Kraken,raw_buf,raw_len,comp_buf,
OodleLZ_CompressionLevel_Optimal1);
where comp_buf should be at least OodleLZ_GetCompressedBufferSizeNeeded bytes large to take
the compressed data.
Here we have chosen the compressor OodleLZ_Compressor_Kraken , which is generally a good balance
of compression ratio and decode speed. We've also chosen a high compression
level (OodleLZ_CompressionLevel_Optimal1), so this encode will be rather slow, but give us a smaller
output size. The compression level
sets the amount of effort the encoder does for string matching, and is how you can control the
encode time (try OodleLZ_CompressionLevel_Fast for example). The compression level does not affect
the decode time, generally.
You can then decompress in a similar way :
SINTa dec_len = OodleLZ_Decompress(comp_buf,comp_len,dec_buf,raw_len);
ASSERT( dec_len == raw_len );
OodleLZ_Decompress will decode data from any of the compressors. Raw OodleLZ compressed data is
headerless - you have to pass in the raw and compressed data sizes to the decode call.
See About OodleLZ for more details about OodleLZ data.
Oodle Core requires no shutdown.
You can see a full working example in example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core.
A note on evaluating Oodle
When evaluating Oodle, be careful what kind of data you use to test the compressors.
You should not run the compressors on already-compressed data such as PNG or JPEG images or Bink movies.
No compressor can do much with already-compressed data, so the differences between them will be hidden.
If you have your game data in an existing archive like a Zip, unpack that before trying
those files in Oodle.
Oodle is optimized for typical game data - levels, meshes, animations, bitmap and hardware compressed textures.
To use Oodle with Granny animation data, make sure you disable Granny's built-in compressor on those files.
In general, on signal media such as images, audio and video, the appropriate data-specific compressor should be used.
On other data that have optional compression, it should be turned off. For example things like pdf or geometry mesh
formats that have optional zlib compression, it's better to turn that compression off and give the data uncompressed
to Oodle, because Oodle will do a better job.
If your data is not compressing as you hoped, feel free to contact oodle@radgametools.com and we'll have
a look at it.
Also, if you want to donate some data to go in our test set, that would be appreciated, and the
Oodle compressors will get better at your data as we refine them.
See some tips on benchmarking here : Tips for benchmarking a compressor
Oodle does not provide packaging or archiving. You create your packages however you want for you
game, and use Oodle just for the data compression.
OodleX provides multi-threaded encode and decode. You can also write your own parallel encode and
decode using Oodle Core with your own threading system.
Oodle currently provides only generic lossless data compression, no specialized image or audio
compression.
The Oodle LZ compressors can encode & decode incrementally, they can encode & decode multi-threaded,
they can provide seek points for random access in compressed streams, they can overlap IO and encoding
or decoding. There are lots of capabilities as you dig deeper.
You can examine some of the more advanced LZ Examples , or visit About OodleLZ or Frequently Asked Questions,
or contact oodle@radgametools.com.
For a guide to how to build with the Oodle library on your platform, see About Oodle on Platforms.
For information about the Core vs Ext libs see Oodle2 Core vs Oodle2 Ext.
| OODLELZ_SPACESPEEDTRADEOFFBYTES_DEFAULT |
|
|
Discussion
Default value of spaceSpeedTradeoffBytes in OodleLZ_CompressOptions
Changes how the encoder makes decisions in the bit stream
Higher spaceSpeedTradeoffBytes favors decode speed more (larger compressed files)
Lower spaceSpeedTradeoffBytes favors smaller compressed files (slower decoder)
Goes in a power of 2 scale; so try 64,128 and 512,1024
(OODLELZ_SPACESPEEDTRADEOFFBYTES_DEFAULT/2) or (OODLELZ_SPACESPEEDTRADEOFFBYTES_DEFAULT*2)
Discussion
Semaphore ; initialize with = 0 , no cleanup necessaryDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
Discussion
Get defaults for OodleXInitOptionsParameters
Return Value
Discussion
The debugSystems and threads options are just easy ways of getting pOptions filled out for common
use cases. For fine control of individual settings, you can always set the values in OodleXInitOptions yourself.
NOTE : do not use this if you want minimal linkage. See OodleX_Init_GetDefaults_Minimal.
Discussion
| OodleXIOQ_GetFileSize_AsyncAndWait |
|
|
Discussion
Convenience version of OodleXIOQ_GetInfoByName_AsyncAndWait
returns negative for error
| OodleXIOQ_CloseFile_Async |
|
|
Discussion
Start a close-file requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
CloseFile also copies any errors on the file to the request, so that an OodleXIOQ_GetStatus on the CloseFile
request will return OodleXStatus_Error if there are any errors on the file.
If the file was OpenForWrite, then truncateFileSize can be used to set the final file size. This is mainly
used when the file was reserved with OodleXIOQ_ReserveFileSizeForWrite_Async , but it should also be used any time
a file size that is not OODLEX_IO_MAX_ALIGNMENT aligned is desired. truncateFileSize does not need to be
OODLEX_IO_MAX_ALIGNMENT aligned, but all sized for OodleXIOQ_Write_Async do, so without doing this file sizes will
be aligned up. Pass OODLEX_FILE_CLOSE_NO_TRUNCATE_SIZE (or use the default argument) if you don't want to truncate.
Discussion
Start a write requestParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Starts an async write from memory at file offset _position , of _size bytes.
The write is not done when OodleXIOQ_Write_Async returns. You must not free memory until the write is done, as reported by
the handle returned.
Writes are faster on some platforms if the file size is first reserved past the end of the write, using
OodleXIOQ_SetFileSize_Async or OodleXIOQ_ReserveFileSizeForWrite_Async.
Discussion
Start a delete request.Parameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
Asynchronously delete a file or dir.
Use OodleXIOQ_ForceWriteable_Async before the Delete to force the deletion of read-only and other no-access conditions.
| OodleXIOQ_WaitDoneAllPending |
|
|
Discussion
Block the calling thread until all pending IOQ operations are completeDiscussion
Should generally only be used for errors or shutdown.
OodleX_WaitDoneAllPending does this and more.
Discussion
Block the calling thread until request is not pendingParameters
| req | the IOQ operation handle to work on
|
| andDelete | if true, delete the request
|
| pErrorCode | (optional) filled with the os error code, if any
|
Return Value
Discussion
The status returned will not be OodleAsync_Pending.
Similar to OodleX_Wait , but only works on IOQ requests, and can return the IOQ error code.
Generally you should just call OodleX_Wait in most cases.
Discussion
Set the global log stateParameters
Discussion
SetState(0) disables all logging. Combine the bit mask flags of OodleXLog_StateFlags
with logical OR to create a log state.
Discussion
Fill a OodleNetwork1TCP_State from training dataParameters
Discussion
OodleNetwork1TCP_Train uses the provided training packet data to initialize state.
The training packet data provided here should not overlap the window passed to OodleNetwork1_Shared_SetWindow ; it should not come from the same source or you will get false training.
You may call OodleNetwork1_Shared_SetWindow and OodleNetwork1TCP_Train many times with different windows to optimize the window selection.
Once training is done, the resulting OodleNetwork1TCP_State should be written to disk and used by both the client and server as the initial channel state in OodleNetwork1TCP_State_InitAsCopy.
| FAQ: What are the speeds and ratios of the OodleLZ compressors and levels? |
|
|
To test the various options for compressor & level, the best option is to run example_lz_chart : Example that makes a chart of OodleLZ options.
This lets you generate numbers on your own machine, on your own file.
In general, OodleLZ_CompressionLevel always trades off worse encode speed for compression ratio. It's the amount of
encoder effort, and typically doesn't affect decode speed much (though better compression also leads to better decode speed).
Here is a sample run of example_lz_chart
showing the choice of OodleLZ_Compressor vs OodleLZ_CompressionLevel. See also FAQ: Which OodleLZ should I use?
and About OodleLZ.
Oodle 2.6.3 example_lz_chart <file>
lz_chart loading r:testsetslztestsetlzt99...
file size : 24700820
Selkie : super fast to encode & decode, least compression
Mermaid: fast decode with better-than-zlib compression
Kraken : good compression, fast decoding, great tradeoff!
Leviathan : very high compression, slowest decode
chart cell shows | raw/comp ratio : encode MB/s : decode MB/s |
All compressors run at various encoder effort levels (SuperFast - Optimal).
Many repetitions are run for accurate timing.
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie |1.41:675:3895|1.45:622:3888|1.53:465:3696|1.68:369:3785|1.70:342:3759|
Mermaid|1.66:436:2189|1.66:436:2188|1.79:352:2090|2.01:276:2055|2.04:261:2025|
Kraken |1.55:588:1839|1.71:419:1136|1.88:331:1087|2.10:279:1093|2.27:167:1010|
compression ratio (raw/comp):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 1.412 | 1.447 | 1.526 | 1.678 | 1.698 |
Mermaid| 1.660 | 1.660 | 1.793 | 2.011 | 2.041 |
Kraken | 1.548 | 1.711 | 1.877 | 2.103 | 2.268 |
encode speed (MB/s):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 674.548 | 621.811 | 464.555 | 369.364 | 341.588 |
Mermaid| 435.650 | 435.923 | 352.475 | 276.199 | 260.511 |
Kraken | 588.488 | 418.921 | 331.423 | 279.129 | 167.206 |
decode speed (MB/s):
| HyperFast4| HyperFast3| HyperFast2| HyperFast1| SuperFast |
Selkie | 3894.644 | 3887.820 | 3695.984 | 3785.457 | 3758.594 |
Mermaid| 2189.030 | 2187.863 | 2090.319 | 2054.897 | 2024.692 |
Kraken | 1839.091 | 1135.920 | 1086.922 | 1093.407 | 1009.967 |
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie |1.75:205:3490|1.83:105:3687|1.86: 43:3815|1.93:5.1:3858|1.94:2.6:3856|
Mermaid|2.12:173:1991|2.19: 84:2177|2.21: 32:2291|2.37:2.8:2058|2.44:1.8:1978|
Kraken |2.32:112:1104|2.39: 37:1187|2.43: 20:1189|2.55:3.1:1103|2.65:1.2:1038|
Leviath|2.50: 31: 738|2.57: 17: 787|2.62:9.5: 807|2.71:1.6: 811|2.76:0.9: 776|
compression ratio (raw/comp):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 1.748 | 1.833 | 1.863 | 1.933 | 1.943 |
Mermaid| 2.118 | 2.194 | 2.207 | 2.367 | 2.437 |
Kraken | 2.320 | 2.390 | 2.434 | 2.551 | 2.646 |
Leviath| 2.504 | 2.572 | 2.617 | 2.707 | 2.756 |
encode speed (MB/s):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 204.621 | 104.758 | 42.504 | 5.102 | 2.554 |
Mermaid| 172.681 | 84.227 | 32.030 | 2.798 | 1.836 |
Kraken | 111.858 | 37.126 | 19.859 | 3.091 | 1.204 |
Leviath| 31.031 | 16.697 | 9.461 | 1.621 | 0.869 |
decode speed (MB/s):
| VeryFast | Fast | Normal | Optimal1 | Optimal3 |
Selkie | 3490.442 | 3686.689 | 3814.655 | 3857.857 | 3856.226 |
Mermaid| 1991.442 | 2176.725 | 2291.498 | 2057.575 | 1977.721 |
Kraken | 1104.172 | 1186.638 | 1189.372 | 1103.148 | 1038.352 |
Leviath| 737.934 | 787.152 | 806.523 | 811.161 | 775.800 |
enum OodleLZ_PackedRawOverlap
{
OodleLZ_PackedRawOverlap_No = 0,
OodleLZ_PackedRawOverlap_Yes = 1,
OodleLZ_PackedRawOverlap_Force32 = 0x40000000
};
Discussion
Bool enum
| OodleLZ_GetSeekEntryPackedPos |
|
|
Discussion
Get the compressed position of a seek entryParameters
Return Value
| return | compressed buffer position of the start of this seek entry
|
Discussion
| OodleNet_Plugins_SetPrintf |
|
|
Discussion
Install the callback used by Oodle Core for loggingParameters
| fp_rrRawPrintf | function pointer to your log function; may be NULL to disable all logging
|
Return Value
| return | returns the previous function pointer
|
Discussion
Use this function to install your own printf for Oodle Core.
The default implementation in debug builds, if you install nothing, uses the C stdio printf for logging.
On Microsoft platforms, it uses OutputDebugString and not stdio.
To disable all logging, call OodleNet_Plugins_SetPrintf(NULL)
WARNING : this function is NOT thread safe! It should be done only once and done in a place where the caller can guarantee thread safety.
In the debug build of Oodle, you can install OodleNet_Plugin_Printf_Verbose to get more verbose logging
Discussion
Set the OodleXLogCallbackRet used for logsParameters
Discussion
The callback is only called if the OODLEXLOG_CALLBACK bit is set in
OodleXLog_SetState
| OodleLZ_Compressor_CanDecodeInCircularWindow |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor can be decoded using a fixed size circular window.
deprecated as of 2.9.0
| OodleLZ_GetAllChunksCompressor |
|
|
Discussion
ask who compressed all chunks in this buf chunkParameters
Return Value
Discussion
returns a simple compressor (for example OodleLZ_Compressor_Kraken) if that was used on all chunks
returns OodleLZ_Compressor_Hydra if different NewLZ encoders were used (for example Kraken+Mermaid)
returns OodleLZ_Compressor_Count if a heterogenous mix of compressors was used (not just NewLZ)
returns OodleLZ_Compressor_Invalid on error
note this is only for this chunk - later chunks may have different compressors (eg. with Hydra)
if you compressed all chunks the same it's up to you to store that info in your header
returns OodleLZ_Compressor_Invalid if compBufSize is too small or any chunk is corrupt
| t_fp_OodleNet_Plugin_Free |
|
|
Discussion
Function pointer type for OodleFreeReturn Value
| return | pointer to memory to free
|
Discussion
| t_fp_OodleNet_Plugin_MallocAligned |
|
|
Discussion
Function pointer type for OodleMallocAlignedParameters
| bytes | number of bytes to allocate
|
| alignment | required alignment of returned pointer
|
Return Value
| return | pointer to memory allocated (must not be NULL)
|
Discussion
alignment will always be a power of two
alignment will always be >= OODLE_MALLOC_MINIMUM_ALIGNMENT
Discussion
Align up to OODLEX_IO_MAX_ALIGNMENTParameters
Return Value
Discussion
Align x up to OODLEX_IO_MAX_ALIGNMENT
Discussion
alloc a large block with "Big" alignmentParameters
| bytes | the amount to allocate (must be > 0)
|
Return Value
| return | pointer to allocated memory
|
Discussion
query the alignment via OodleXMallocBigAlignment
Discussion
Enables or disables Oodle usage warnings.Parameters
| state | whether usage warnings should be enabled or disabled.
|
Discussion
Usage warnings are enabled by default and try to be low-noise, but in case you want to
disable them, this is how.
This should generally be done once at startup. Setting this state while there are Oodle
calls running on other threads has undefined results.
Discussion
Encode a packetParameters
Return Value
| return | length of output compressed data written to comp ; the returned compLen is strictly <= rawLen
|
Discussion
Encodes one packet.
state and shared are both const and can be shared by all encoders and decoders.
The returned compLen will never be greater than rawLen, because OodleNetwork1 won't send packets that expand under compression (it just sends them uncompressed) - however it may write further than that during the compression attempt. Do not use the returned compLen to check the size of the compressed buffer needed.
| Oodle Network and Data SDK separation |
|
|
As of Oodle 2.7.0 , Oodle Network is now a separate SDK from Oodle Data.
Oodle Data compression provides two libraries (Core and Ext). Oodle Network provides another library.
For example on Windows you have :
In the Data SDK :
oo2core_win64.lib
oo2ext_win64.lib
In the Network SDK :
oo2net_win64.lib
Oodle Data & Network are independent and do not link together.
If you are using Network & Data of the same SDK version, you may install the SDKs to the same place, allow
them to overwrite shared files that are identical in the two SDKs.
The Oodle Network examples build without Oodle Data, but can optionally be set to use Oodle Data to compress
the shared model state.
Oodle Core and Oodle Network have similar but independent plugins that the client can set to control their
interaction with the system via logging, allocations, and assertions.
If you use both SDKs, you can install your functions by calling both plugins, for example OodleNet_Plugins_SetAllocators and OodleCore_Plugins_SetAllocators.
If you do Oodle Ext Init, the Oodle Ext plugins are installed to Oodle Core. Oodle Ext does not interact with
Oodle Network.
struct OodleXInitOptions
{
const OodleXMallocVTable * m_pBaseVTable;
OO_BOOL m_OodleInit_DebugAllocator;
OO_S32 m_num_handles_log2;
OO_BOOL m_OodleInit_ThreadLog;
OO_BOOL m_OodleInit_Log;
OO_BOOL m_OodleInit_Log_Header;
const char * m_OodleInit_Log_FileName;
OO_BOOL m_OodleInit_Log_FlushEachWrite;
OO_BOOL m_OodleInit_BreakOnLogError;
OO_BOOL m_OodleInit_Telemetry;
void * m_OodleInit_Telemetry_Context;
OO_BOOL m_OodleInit_StackTrace;
OO_BOOL m_OodleInit_LeakTrack;
OO_BOOL m_OodleInit_SimpleProf;
OO_BOOL m_OodleInit_FuzzTest;
t_OodleFPVoidVoid * m_OodleInit_ThreadProfiler_funcptr;
OO_BOOL m_OodleInit_IOQ;
OO_BOOL m_OodleInit_IOQ_Log;
OO_BOOL m_OodleInit_IOQ_BreakOnError;
OO_BOOL m_OodleInit_IOQ_Threaded;
OO_BOOL m_OodleInit_IOQ_CheckAlignment;
OO_BOOL m_OodleInit_Workers;
OO_S32 m_OodleInit_Workers_Count;
};
Discussion
Options struct for OodleX_Init
can be filled with OodleX_Init_GetDefaults
Members
Discussion
Block the calling thread until handle is not PendingParameters
Return Value
Discussion
Will not return OodleXStatus_Pending.
OodleX_WaitNoDelete and OodleX_WaitAndDelete are macros that are provided as short-hands for OodleX_Wait. They are :
#define OodleX_WaitNoDelete (h) OodleX_Wait(h,OodleXHandleDeleteIfDone_No)
#define OodleX_WaitAndDelete(h) OodleX_Wait(h,OodleXHandleDeleteIfDone_Yes)
| example_network_client : Example with simple network client support |
|
|
Discussion
Oodle example_network_client
This is an example of an Oodle Network UDP client (such as in a game's runtime).
It loads a previously trained model, such as created by example_packet : Example demonstrating network packet compression. This would be done offline by your game's
tools.
The runtime component loads a previously created model and uses it to compress packets on the fly.
This example uses only Oodle2 Network lib.
(Oodle2 data compression is optional)
#include "../include/oodle2net.h"
// optional, use Oodle LZ for the network model data
//#define USE_OODLE_LZ_DATA_COMPRESSION
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
#include "../include/oodle2.h"
#endif
#include "ooex.h" // example helpers
#ifndef _CRT_SECURE_NO_WARNINGS
#define _CRT_SECURE_NO_WARNINGS
#endif
#include <stdio.h>
#include <string.h>
#include <assert.h>
#include <stdlib.h>
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_network_client
#endif
#include "read_whole_file.h"
example_packet_file should contain a capture of packets from a real game session.
It should be at least 100 MB.
See Capturing Training data for OodleNetwork
Some portion (as set by example_packet_test_holdout_fraction_denominator) will be held out for testing
the compression level. The remainder will be used for training. In real game usage, you don't need to
hold out any for testing, this is just for evaluating the compression level. To be fair, the packets
used in training are not used in testing.
The file format is :
packet.bin :
OO_U32 [LE] : numbers of channels (num_channels)
repeatedly :
{
OO_U32 [LE] : channel index in [0,num_channels-1]
OO_U32 [LE] : number of bytes of data in this packet (num_bytes)
OO_U8 * num_bytes : payload of this packet
}
static const char * c_example_packet_file = "r:\\packet.bin"; // <- change this to your file name
// runtimedata file name written by "example_packet" :
static const char * c_runtimedata_fileName = "example_packet_on1udpnew_runtimedata.bin";
// OodleNetworkRuntimeData is held by the client & server for runtime encoding or decoding
struct OodleNetworkRuntimeData;
// load a previously saved network compression model :
static OodleNetworkRuntimeData * OodleNetwork_CreateFromFile( const char * fileName );
// release the compressor memory :
static void OodleNetwork_Destroy(OodleNetworkRuntimeData * pCompressor);
// OodleNetwork_Encode returns compressed size
static OO_SINTa OodleNetwork_Encode(
const OodleNetworkRuntimeData * c,
const void * raw, OO_SINTa rawLen,
void * comp );
// OodleNetwork_Decode returned decompressed size, or 0 for error
static OO_SINTa OodleNetwork_Decode(
const OodleNetworkRuntimeData * c,
const void * comp, OO_SINTa compLen,
void * raw );
extern "C" int main(int argc,char *argv[])
{
printf("example_network_client [packet_file] [runtimedata_file]\n");
// defaults :
const char * example_packet_file = c_example_packet_file;
const char * runtimedata_file = c_runtimedata_fileName;
// override from command line args :
if ( argc >= 2 )
{
example_packet_file = argv[1];
if ( argc >= 3 )
runtimedata_file = argv[2];
}
---------------------------------------------------------------------
Load the previously trained model data
This compressor is const and will be used to compress all packets
---------------------------------------------------
printf("Loading runtimedata file : %s\n",runtimedata_file);
OodleNetworkRuntimeData * c = OodleNetwork_CreateFromFile( runtimedata_file );
if ( c == NULL )
{
printf("FAILED!\n");
return 10;
}
-----------------------------------------------------
Load some test packet data from a file
-----------------------------------
printf("Loading packet file : %s\n",example_packet_file);
void * packet_file_data;
OO_SINTa packet_file_size;
packet_file_data = read_whole_file(example_packet_file,&packet_file_size);
if ( packet_file_data == NULL )
{
printf("ERROR : couldn't load packet file\n");
OodleNetwork_Destroy(c);
return 10;
}
-----------------------------------------------------
Use the OodleNetwork model to encode & decode the packets :
--------------------------------------
printf("Testing compression...\n");
// point at the packet data :
const OO_U8 * packet_bin_buf = (const OO_U8 *) packet_file_data;
OO_SINTa packet_bin_size = packet_file_size;
// get the number of channels ; first dword in packet bin :
const OO_U8 * packet_bin_ptr = packet_bin_buf;
const OO_U8 * packet_bin_end = packet_bin_ptr + packet_bin_size;
OO_S32 num_channels = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
// num_channels ignored for UDP
OOEX_UNUSED_VARIABLE(num_channels);
// max packet size is just used for the stack buffer sizes :
#define MAX_PACKET_SIZE 16384
// technically comp_buffer should be OodleNetwork1_CompressedBufferSizeNeeded(MAX_PACKET_SIZE) bytes
OO_U8 comp_buffer[MAX_PACKET_SIZE];
OO_U8 decomp_buffer[MAX_PACKET_SIZE];
OO_S64 tot_rawLen = 0;
OO_S64 tot_compLen = 0;
OO_S64 tot_numPackets = 0;
while( packet_bin_ptr < packet_bin_end )
{
// grab the current packet header :
OO_S32 channel = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_UNUSED_VARIABLE(channel);
OO_S32 bytes = OOEX_GET32_LE(packet_bin_ptr); packet_bin_ptr += sizeof(OO_S32);
OOEX_ASSERT( channel >= 0 && channel < num_channels );
OOEX_ASSERT( bytes > 0 );
if ( packet_bin_ptr + bytes > packet_bin_end )
{
// partial end packet
break;
}
OOEX_ASSERT( bytes < MAX_PACKET_SIZE );
// compress one packet : (right before sending over the network)
OO_SINTa cur_compLen =
OodleNetwork_Encode(c,packet_bin_ptr,bytes,comp_buffer);
// [comp_buffer,cur_compLen] is sent over the network
// check I'm not lying about compLen :
comp_buffer[cur_compLen] = 0xCD;
// decompress it : (upon receipt from the net)
OO_SINTa decode_rawLen = OodleNetwork_Decode(c,
comp_buffer,cur_compLen,
decomp_buffer);
OOEX_ASSERT_ALWAYS( decode_rawLen == bytes );
// verify we got the packet back :
OOEX_ASSERT_ALWAYS( memcmp(packet_bin_ptr,decomp_buffer,bytes) == 0 );
packet_bin_ptr += bytes;
tot_rawLen += bytes;
tot_compLen += cur_compLen;
tot_numPackets ++;
}
free(packet_file_data);
printf("%d packets compressed %.2f -> %.2f average\n",(int)tot_numPackets,
(double)tot_rawLen/tot_numPackets,(double)tot_compLen/tot_numPackets);
//-----------------------------------------------------
// release the OodleNetwork memory :
OodleNetwork_Destroy(c);
return 0;
}
=================================================================================
Sample packet size transmission using "encodemod"
Packet size is sent in 1 byte if possible, else 2 bytes
This encoding assumes a max packet size is known.
c_packetsize_modbits should be set as low as possible, while still ensuring c_packetsize_max
is large enough to send the largest packet size.
For example if your max packet size is 1500 , you could use modbits=3
static const int c_packetsize_modbits = 6;
static const int c_packetsize_immediate = 256 - (1<<c_packetsize_modbits); // = 192 for modbits=6
static const int c_packetsize_max = c_packetsize_immediate + (1<<(c_packetsize_modbits+8)); // =16576 for modbits = 6
static OO_U8 * PutPacketSize(OO_U8 * ptr,OO_SINTa size)
{
OOEX_ASSERT( size < c_packetsize_max );
if ( size < c_packetsize_immediate )
{
*ptr++ = (OO_U8) size;
}
else
{
size -= c_packetsize_immediate;
OO_U8 lo = (OO_U8) size;
OO_U8 hi = (OO_U8) (size>>8);
OOEX_ASSERT( hi < 256-c_packetsize_immediate );
*ptr++ = c_packetsize_immediate + hi;
*ptr++ = lo;
}
return ptr;
}
static const OO_U8 * GetPacketSize(const OO_U8 * ptr,OO_SINTa * psize)
{
OO_U8 first = *ptr++;
if ( first < c_packetsize_immediate )
{
*psize = first;
}
else
{
int hi = first - c_packetsize_immediate;
int lo = *ptr++;
*psize = (hi<<8) + lo + c_packetsize_immediate;
}
return ptr;
}
//=================================================================================
OodleNetwork1_SavedModel_Header is an example header for the saved Oodle Network trained model.
This is copied from example_packet
You could of course use your own header, and your own engine's IO code to persist this data.
struct OodleNetwork1_SavedModel_Header
{
#define ON1_MAGIC 0x11235801
OO_U32 magic;
OO_U32 compressor;
OO_U32 ht_bits;
OO_U32 dic_size;
OO_U32 oodle_major_version;
OO_U32 dic_complen;
OO_U32 statecompacted_size;
OO_U32 statecompacted_complen;
};
OodleNetwork1_SavedModel_Header is written to file like a flat struct,
but we ensure it's always little endian.
static void OodleNetwork1_SavedModel_Header_Read(OodleNetwork1_SavedModel_Header * pHeader,const void * from_memory)
{
const OO_U32 * from_ptr = (const OO_U32 *)from_memory;
pHeader->magic = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->compressor = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->ht_bits = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->dic_size = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->oodle_major_version = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->dic_complen = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->statecompacted_size = OOEX_GET32_LE(from_ptr); from_ptr++;
pHeader->statecompacted_complen = OOEX_GET32_LE(from_ptr); from_ptr++;
OOEX_ASSERT( (((OO_U8 *)from_ptr) - ((OO_U8 *)from_memory)) == sizeof(OodleNetwork1_SavedModel_Header) );
}
struct OodleNetworkRuntimeData
{
void * dic;
OodleNetwork1UDP_State * state;
OodleNetwork1_Shared * shared;
};
static OO_SINTa OodleNetwork_Encode(
const OodleNetworkRuntimeData * c,
const void * raw, OO_SINTa rawLen,
void * comp )
{
// put uncompressed packet size first :
// NOTE : if you have this in some kind of header already, then don't put it again here!
OO_U8 * compPtr = (OO_U8 *)comp;
compPtr = PutPacketSize(compPtr,rawLen);
// then put the compressed packet :
compPtr += OodleNetwork1UDP_Encode(c->state,c->shared,raw,rawLen,compPtr);
return compPtr - (OO_U8 *)comp;
}
static OO_SINTa OodleNetwork_Decode(
const OodleNetworkRuntimeData * c,
const void * comp, OO_SINTa compLen,
void * raw )
{
const OO_U8 * compPtr = (const OO_U8 * )comp;
OO_SINTa rawLen;
// first get the uncompressed packet size :
compPtr = GetPacketSize(compPtr,&rawLen);
OO_SINTa packet_compLen = ((const OO_U8 * )comp + compLen) - compPtr;
// try to check that the packet size we got makes sense :
// it should be >= compressed size because OodleNetwork never expands :
OOEX_ASSERT( rawLen >= packet_compLen );
if ( rawLen < packet_compLen )
return 0; // corruption!
// then decode the packet :
OO_BOOL ok = OodleNetwork1UDP_Decode(c->state,c->shared,compPtr,packet_compLen,
raw,rawLen);
if ( ! ok )
return 0;
return rawLen;
}
static void OodleNetwork_Destroy(OodleNetworkRuntimeData * pCompressor)
{
free(pCompressor->dic);
free(pCompressor->shared);
free(pCompressor->state);
free(pCompressor);
}
static bool OodleNetwork_LoadFromFileData( OodleNetworkRuntimeData * pCompressor, const void * fileData, OO_SINTa fileSize )
{
// parse header :
OodleNetwork1_SavedModel_Header header;
OodleNetwork1_SavedModel_Header_Read(&header,fileData);
if ( header.magic != ON1_MAGIC )
{
printf("ERROR : state_file ON1_MAGIC mismatch\n");
return false;
}
#if 0
// optional check for oodle_major_version being the same
// NOTE : this is more conservative than necessary
// you may disable this check to keep loading old state files
OO_U32 oodle_major_version_major = header.oodle_major_version;
if ( oodle_major_version_major < 1 )
{
OodleXLog_Printf_v0("state_file version too old!\n");
return false;
}
if ( oodle_major_version_major > OODLE2NET_VERSION_MAJOR )
{
OodleXLog_Printf_v0("state_file version newer that SDK!\n");
return false;
}
#endif
OO_S32 on1udpnew_ht_bits = header.ht_bits;
OO_SINTa on1udpnew_dic_size = header.dic_size;
OO_SINTa on1udpnew_dic_complen = header.dic_complen;
OO_SINTa on1udpnew_statecompacted_size = header.statecompacted_size;
OO_SINTa on1udpnew_statecompacted_complen = header.statecompacted_complen;
OOEX_ASSERT_ALWAYS( on1udpnew_dic_size >= on1udpnew_dic_complen );
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size >= on1udpnew_statecompacted_complen );
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size > 0 && on1udpnew_statecompacted_size < OodleNetwork1UDP_StateCompacted_MaxSize() );
OOEX_ASSERT_ALWAYS( fileSize == (OO_S64)sizeof(OodleNetwork1_SavedModel_Header) + on1udpnew_dic_complen + on1udpnew_statecompacted_complen );
// printf("OodleNetwork1UDP Loading; dic comp %d , state %d->%d\n",
// (int)on1udpnew_dic_complen,(int)on1udpnew_statecompacted_size,(int)on1udpnew_statecompacted_complen);
//-------------------------------------------
// decompress on1udpnew_dic and on1udpnew_statecompacted
pCompressor->dic = malloc(on1udpnew_dic_size);
const void * on1udpnew_dic_comp_ptr = (const OO_U8 *)(fileData) + sizeof(OodleNetwork1_SavedModel_Header);
OodleNetwork1UDP_StateCompacted * on1udpnew_compacted = (OodleNetwork1UDP_StateCompacted *) malloc( on1udpnew_statecompacted_size );
void * on1udpnew_statecompacted_comp_ptr = (OO_U8 *)on1udpnew_dic_comp_ptr + on1udpnew_dic_complen;
#ifdef USE_OODLE_LZ_DATA_COMPRESSION
// the dictionary and the state might be compressed with Oodle LZ
// decode them :
if ( header.compressor == (OO_U32)OodleLZ_Compressor_Invalid )
{
OOEX_ASSERT_ALWAYS( on1udpnew_dic_complen == on1udpnew_dic_size );
memcpy(pCompressor->dic,on1udpnew_dic_comp_ptr,on1udpnew_dic_size);
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size == on1udpnew_statecompacted_complen );
memcpy(on1udpnew_compacted,on1udpnew_statecompacted_comp_ptr,on1udpnew_statecompacted_size);
}
else
{
OO_SINTa decomp_dic_size = OodleLZ_Decompress(on1udpnew_dic_comp_ptr,on1udpnew_dic_complen,pCompressor->dic,on1udpnew_dic_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( decomp_dic_size == on1udpnew_dic_size );
OO_SINTa decomp_statecompacted_size = OodleLZ_Decompress(on1udpnew_statecompacted_comp_ptr,on1udpnew_statecompacted_complen,on1udpnew_compacted,on1udpnew_statecompacted_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( decomp_statecompacted_size == on1udpnew_statecompacted_size );
}
#else
OOEX_ASSERT_ALWAYS( header.compressor == (OO_U32)-1 );
OOEX_ASSERT_ALWAYS( on1udpnew_dic_complen == on1udpnew_dic_size );
memcpy(pCompressor->dic,on1udpnew_dic_comp_ptr,on1udpnew_dic_size);
OOEX_ASSERT_ALWAYS( on1udpnew_statecompacted_size == on1udpnew_statecompacted_complen );
memcpy(on1udpnew_compacted,on1udpnew_statecompacted_comp_ptr,on1udpnew_statecompacted_size);
#endif
//OodleXFree_IOAligned(state_fileData); state_fileData = NULL;
//----------------------------------------------
// Uncompact the "Compacted" state into a usable state
OO_SINTa on1udpnew_state_size = OodleNetwork1UDP_State_Size();
pCompressor->state = (OodleNetwork1UDP_State *) malloc( on1udpnew_state_size );
OOEX_ASSERT_ALWAYS( pCompressor->state != NULL );
if ( ! OodleNetwork1UDP_State_Uncompact(pCompressor->state,on1udpnew_compacted) )
{
printf("OodleNetwork1UDP_State_Uncompact failed\n");
return false;
}
free(on1udpnew_compacted);
//----------------------------------------------
// fill out on1udpnew_shared from the dictionary
OO_SINTa shared_size = OodleNetwork1_Shared_Size(header.ht_bits);
pCompressor->shared = (OodleNetwork1_Shared *) malloc( shared_size );
OOEX_ASSERT_ALWAYS( pCompressor->shared != NULL );
OodleNetwork1_Shared_SetWindow(pCompressor->shared,on1udpnew_ht_bits,pCompressor->dic,(OO_S32)on1udpnew_dic_size);
return true;
}
static OodleNetworkRuntimeData * OodleNetwork_CreateFromFile( const char * fileName )
{
void * fileData;
OO_SINTa fileSize;
fileData = read_whole_file(fileName,&fileSize);
if ( ! fileData )
{
printf("ERROR : couldn't read file %s\n",fileName);
return NULL;
}
OodleNetworkRuntimeData * c = (OodleNetworkRuntimeData *) malloc( sizeof(OodleNetworkRuntimeData) );
memset(c,0,sizeof(*c));
bool ok = OodleNetwork_LoadFromFileData(c,fileData,fileSize);
free(fileData);
if ( ! ok )
{
printf("ERROR : couldn't parse file %s\n",fileName);
OodleNetwork_Destroy(c);
return NULL;
}
return c;
}
//==================================================================================
| OodleCore_Plugins_SetJobSystemAndCount |
|
|
Discussion
Set the function pointers for async job system needed by Oodle2 CoreParameters
Discussion
If these are not set, the default implementation runs jobs synchronously on the calling thread.
These must not be changed once they are set! Set them once then don't change them.
target_parallelism allows you to tell Oodle how many Jobs it should try to keep in flight at once.
Depending on the operation it may not be able to split work into this many jobs (so fewer will be used),
but it will not exceed this count.
For Oodle Data LZ work, typically target_parallelism is usually best at the number of hardware cores
not including hyper threads).
For Oodle Texture BCN encoding work, target_parallelism is usually best as the full number of hyper cores.
In some cases you may wish to reduce target_parallelism by 1 or 2 cores to leave some of the CPU free for
other work.
For example on a CPU with 16 cores and 32 hardware threads, for LZ work you might set target_parallelism to 15
when calling OodleCorePlugins. For BC7 encoding you might set target_parallelism to 30 when calling OodleTexPlugins.
NOTE : if you are using Oodle Ext, do NOT call this. OodleX_Init will install a job system for Oodle Core.
Note OodleX only installs automatically to Oodle Core, not Net or Tex. See example_jobify.cpp for manual
plugin.
Replaces deprecated OodleCore_Plugins_SetJobSystem
See About Oodle Job Threading Plugins
| OodleCore_Plugins_SetAllocators |
|
|
Discussion
Set the function pointers for allocation needed by Oodle2 CoreDiscussion
If these are not set, the default implementation on most platforms uses the C stdlib.
On Microsoft platforms the default implementation uses HeapAlloc.
These must not be changed once they are set! Set them once then don't change them.
NOTE: if you are using Oodle Ext, do NOT call this. OodleX_Init will install an allocator for Oodle Core. Do not mix your own allocator with the OodleX allocator. See OodleX Memory Allocators.
If you want to ensure that Oodle is not doing any allocations, you can call OodleCore_Plugins_SetAllocators(NULL,NULL);
If you do that, then any time Oodle needs to allocate memory internally, it will stop the process.
It is STRONGLY not recommended that you ship that way. You can verify that Oodle is not allocating, but then leave some
fallback allocator installed when you actually ship just in case.
Also note that on many consoles the standard allocation practices may not
leave much heap memory for the C stdlib malloc. In this case Oodle may fail to allocate.
| OodleLZ_GetChunkCompressor |
|
|
Discussion
Deprecated entry point for backwards compatibilityDiscussion
Use OodleLZ_GetFirstChunkCompressor or OodleLZ_GetAllChunksCompressor
Discussion
Frees a table allocated by OodleLZ_CreateSeekTable
| t_fp_OodleCore_Plugin_DisplayAssertion |
|
|
Discussion
Function pointer to Oodle Core assert callbackParameters
| file | C file that triggered the assert
|
| line | C line that triggered the assert
|
| function | C function that triggered the assert (may be NULL)
|
| message | assert message
|
Return Value
| return | true to break execution at the assertion site, false to continue
|
Discussion
This callback is called by Oodle Core when it detects an assertion condition.
This will only happen in debug builds.
Discussion
Oodle low level offsets and sizes are aligned to OODLEX_IO_MAX_ALIGNMENT Discussion
Unbuffered IO (as in OodleX low level async io) require alignment to OODLEX_IO_MAX_ALIGNMENT.
Pointers returned by OodleXMalloc_IOAligned are so aligned.
You can also use the utility functions such as OodleX_IOAlignUpS32 to align values.
| OodleLZDecoder_DecodeSome |
|
|
OO_BOOL OodleLZDecoder_DecodeSome( OodleLZDecoder * decoder,
OodleLZ_DecodeSome_Out * out,
void * decBuf,
OO_SINTa decBufPos,
OO_SINTa decBufferSize,
OO_SINTa decBufAvail,
const void * compPtr,
OO_SINTa compAvail,
OodleLZ_FuzzSafe fuzzSafe OODEFAULT( OodleLZ_FuzzSafe_No ),
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
OodleLZ_Decode_ThreadPhase threadPhase OODEFAULT( OodleLZ_Decode_Unthreaded ) );Discussion
Incremental decode some LZ compressed dataParameters
Return Value
| return | true if success, false if invalid arguments or data is encountered
|
Discussion
Decodes data encoded with an OodleLZ compressor.
Decodes an integer number of quanta; quanta are OODLELZ_QUANTUM_LEN uncompressed bytes.
decBuf can either be a circular window or the whole rawLen array.
In either case, decBufPos should be in the range [0,decBufferSize).
If decBuf is a circular window, then decBufferSize should come from OodleLZDecoder_MakeValidCircularWindowSize.
(circular windows are deprecated as of 2.9.0)
NOTE : all the new LZ codecs (Kraken, etc.) do not do circular windows. They can do sliding windows, see lz_test_11 in example_lz : Example demonstrating LZ compression and decompression.
They should always have decBufferSize = total raw size, even if the decode buffer is smaller than that.
NOTE : insufficient data provided (with compAvail > 0 but not enough to decode a quantum) is a success case
(return value of true), even though nothing is decoded. A return of false always indicates a non-recoverable error.
If decBufAvail or compAvail is insufficient for any decompression, the "curQuantum" fields of OodleLZ_DecodeSome_Out
will tell you how much you must provide to proceed. That is, if enough compressed bytes are provided to get a quantum header, but not enough to decode a quantum, this
function returns true and fills out the OodleLZ_DecodeSome_Out structure with the size of the quantum.
See OodleLZ_Decompress about fuzz safety.
NOTE : DecodeSome expect to decode either one full quantum (of len OODLELZ_QUANTUM_LEN) or up to the length of the total buffer specified in the
call to OodleLZDecoder_Create or OodleLZDecoder_Reset. That total buffer length
must match what was use during compression (or be a seek-chunk portion thereof).
That is, you cannot decompress partial streams in intervals smaller than
OODLELZ_QUANTUM_LEN except for the final partial quantum at the end of the stream.
| OodleNetwork1UDP_State_Uncompact |
|
|
Discussion
Fills a OodleNetwork1UDP_State from a OodleNetwork1UDP_StateCompactedParameters
Return Value
| return | false if invalid data is detected
|
Discussion
Use this when the OodleNetwork1UDP_StateCompacted is read from a file, so that the OodleNetwork1UDP_State can be used for coding. You may discard the OodleNetwork1UDP_StateCompacted after this call.
to should be allocated to OodleNetwork1UDP_State_Size bytes.
NOTE : the return value here is not a robust way to detect tampered data. You should use encryption and/or a safe checksum on your saved model file data.
Discussion
Fill a OodleNetwork1UDP_State from training dataParameters
Discussion
OodleNetwork1UDP_Train uses the provided training packet data to initialize state.
The training packet data provided here should not overlap the window passed to OodleNetwork1_Shared_SetWindow ; it should not come from the same source or you will get false training.
You may call OodleNetwork1_Shared_SetWindow and OodleNetwork1UDP_Train many times with different windows to optimize the window selection.
Once training is done, the resulting OodleNetwork1UDP_State should be written to disk and used by both the client and server.
There's no need to copy a OodleNetwork1UDP_State , the same state object can be used by all encoders and decoders.
enum OodleXCopyFileFlags
{
OodleXCopyFileFlags_Overwrite = 0,
OodleXCopyFileFlags_DontOverwriteExisting = 1,
OodleXCopyFileFlags_OverwriteOnlyIfNewer = 2,
OodleXCopyFileFlags_OverwriteOnlyIfDifferentSize = 4,
OodleXCopyFileFlags_OverwriteOnlyIfNewerOrDifferentSize = 2|4,
OodleXCopyFileFlags_Mask = 7,
OodleXCopyFileFlags_Default = 0,
OodleXCopyFileFlags_Force32 = 0x40000000
};
Discussion
Flags for Oodle CopyFile operations.Enumerants
Discussion
Combine with logical OR.
| OodleXMalloc_InstallVTable |
|
|
Discussion
Install the vtable that OodleX will use to allocate memoryParameters
| pvt | pointer to the vtable to be installed (will be copied)
|
| pBaseVT | if pvt is a layered vtable, this is the underlying alloc; if not it should be = pvt
|
Discussion
Sets the global vtable that will de used by the OodleXMalloc calls. Typically let OodleX_Init install a
suitable vtable for you. If you do it manually, it must be done before any other OodleX initialization.
WARNING : You must not change the vtable after OodleX is running; pointers allocated from the previous
vtable will still need to be freed and will call to the global vtable.
Discussion
OodleX_Semaphore_PostDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
| OodleXHandleCountdown_Decrement |
|
|
Discussion
Decrement a countdown handle created by OodleXHandleCountdown_AllocParameters
Return Value
| return | status after the decrement
|
Discussion
Returns OodleXStatus_Done if this decrement took the countdown to 0, else OodleXStatus_Pending.
| OodleLZDecoder_MemorySizeNeeded |
|
|
Discussion
If you want to provide the memory needed by OodleLZDecoder_Create , this tells you how big it must be.Parameters
Return Value
| return | bytes to allocate or reserve, 0 for failure
|
Discussion
NOTE : using OodleLZ_Compressor_Invalid lets you decode any time of compressed data.
It requests as much memory as the largest compressor. This may be a lot more than your data needs;
try to use the correct compressor type.
If rawLen is -1 (default) then the Decoder object created can be used on any length of raw data
decompression. If rawLen is specified here, then you can only use it to decode data shorter than
the length you specified here. This use case is very rare, contact support for details.
| FAQ: How do I get the Oodle logs? |
|
|
When you have a problem with Oodle, I might ask you to send me your log file. This is how to find it.
(if you are using Oodle Core only, with no Oodle X, Oodle does not write its own log file, that's up to you.
You could link in OodleX and use it to enable logging. This gets OodleX to call OodlePlugins_SetPrintf to
install its logger in Oodle Core. Another option with Oodle Core is to use OodlePlugins_SetPrintf with
OodlePlugin_Printf_Verbose to get more detailed logging, but only in the debug version of Oodle).
NOTE : if you are using Core only on a Microsoft target, the default core log function only goes to
OutputDebugString - not stdio - so you may not see any error messages unless you are running in the debugger.
Either use Oodle X or install your own logger to get messages however you want.
If you are running Oodle Ext with the default options to OodleX_Init, a log file is written by default.
If you are setting the options by hand, ensure OodleXInitOptions:m_OodleInit_Log is on. You can also
toggle logging on the fly with OodleXLog_SetState.
To get more information in the log, call OodleXLog_SetVerboseLevel with OodleXLog_Verbose_Lots.
You can also pass a higher level of OodleLZ_Verbosity to the OodleLZ_Decompress call.
If you know which function is failing, it helps to call OodleXLog_Printf with the value of the arguments that
you are passing (Oodle does not automatically log the value of every argument passed).
If your app is crashing because of the error and you can't debug, you
can set OodleXInitOptions:m_OodleInit_Log_FlushEachWrite ; hopefully this will
get the log to flush out some information about the error before the crash.
You can set the log file name in OodleXInitOptions:m_OodleInit_Log_FileName. If you don't set it, Oodle will
use a default name that depends on the platform
(see About Oodle on Platforms for per-platform log info).
It may be useful to call OodleX_LogSystemInfo at startup (after calling OodleX_Init).
This will write a bunch of information about your machine and your Oodle build version
to the log file. It is not done automatically.
Please grab the entire log file and mail it to me, don't send just a portion of it.
Discussion
Decode a packetParameters
| state | const shared compression state
|
| shared | const shared compression context
|
| comp | compressed packet received
|
| compLen | size of compressed data
|
| raw | output decompressed packet
|
| rawLen | size of the packet to write
|
Return Value
Discussion
Decodes one packet.
state and shared are both const and can be shared by all encoders and decoders.
The rawLen provided here must match the length used in OodleNetwork1UDP_Encode when creating this compressed packet. The OodleNetwork1 data is headerless, it's up to you to send the packet decompressed size in your own header.
If corrupt data is detected, false is returned.
If the number of compressed bytes consumed does not match compLen, false is returned.
If the number of output bytes does not match rawLen, false is returned.
This function, however, does not do verify data integrity. It will return 'true' if the correct number of bytes are coded,
even if the data does not match.
The buffer comp must be allowed to read at least compLen + OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN bytes
(note that this is strictly less than OodleNetwork1_CompressedBufferSizeNeeded(rawLen) , so that may be used as well)
Discussion
Opaque weak reference handle to an IOQ File
| OodleLZ_ThreadPhased_BlockDecoderMemorySizeNeeded |
|
|
Discussion
Returns the size of the decoder needed for ThreadPhased decodeDiscussion
For use with OodleLZ_Decode_ThreadPhase
See About OodleLZ ThreadPhased Decode
| OodleXIOQ_SetInfoByName_AsyncAndWait |
|
|
Discussion
See OodleXIOQ_SetInfoByName_Async
| OodleX_GetNumWorkerThreads |
|
|
Discussion
Returns the number of worker threadsDiscussion
When there are 0 worker threads, the OodleWork system still succeeds, it just runs Worklets synchronously
on the calling thread.
The worker thread count is set in OodleXInitOptions:m_OodleInit_Workers_Count
| OodleLZ_Compressor_GetName |
|
|
Discussion
Provides a string naming a OodleLZ_Compressor compressor
Discussion
Get current time in secondsReturn Value
| return | current time in seconds
|
Discussion
What it do.
| OodleLZ_Compressor_CanDecodeInPlace |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor can be used with "in-place" decoding.
This is now always true (all compressors support in-place decoding). The function is left
for backward compatibility.
All compressors in the future will support in-place, you don't need to check this property.
| OodleX_Shutdown_DebugBreakOnLeaks |
|
|
enum OodleX_Shutdown_DebugBreakOnLeaks
{
OodleX_Shutdown_DebugBreakOnLeaks_No = 0,
OodleX_Shutdown_DebugBreakOnLeaks_Yes = 1,
OodleX_Shutdown_DebugBreakOnLeaks_Force32 = 0x40000000
};
Discussion
bool enum
| Forming Packets for Maximum Compression |
|
|
The way you create your network packets can have a large effect on their compressability.
These are a few tips for people who are interested in experimenting to get more compression.
What works on different games varies, some of these may not work for your game.
- Try not to include pre-compressed data. Don't use zlib or any other compressor before giving the data to
Oodle. If you do need to send pre-compressed data to the client, don't run those packets through OodleNetwork,
and don't include them in the training set.
- Getting rid of floats is always good. Floating point numbers usually have a lot of useless precision in
the low bits that the game doesn't actually care about. These show up as a nearly random byte in the low
part of the word. It's best to figure out how much precision you actually need and covert all floats to
fixed point integers.
- Do do reductions that are mathematical in nature, which the compressor will never be able to figure out.
For example, send matrices as Euler angles, send only 2 components of normalized vectors, etc.
- Don't worry too much about bit packing or entropy reduction that is purely statistical. This is what the
Oodle Network compressor is good at.
- Consider turning off bit packing entirely. In some cases it's better to just do byte-aligned packet
construction and let the compressor handle all the bit packing. Even though the uncompressed packet will be
much bigger, it can result in a smaller compressed packet. It also may save a lot of CPU time by avoiding the
bit packing. (this varies a lot from game to game)
- If you do bit-pack, try to byte-align in strategic places. At the very least at the start of each game entity
you should flush to byte alignment. Possibly in other places, such as between the property index and the property
value.
- Try to use absolute indices instead of relative indices when refering to properties or other entities.
This makes the identifier stable and consistent, so that the compressor can identify repeated uses of the same
identifier.
- If possible, try to include a hint about the property or entity in its first bytes. Generally don't do this
if requires adding a byte you weren't already sending. For example if you send an entity index, use the top bits
to indicate the type of entity. That way the compressor can see the first bytes and use them to expect what's
coming next.
- Very large and rare packets should usually be compressed with Oodle Data Compression and not piped through the
Oodle Network system. For example you might send a huge initial join packet that's 100 kB , then normal game play
packets are around 200 bytes. Those are very different things and should be compressed separately.
- Split packets for MTU after Oodle network, not before.
See also :
http://cbloomrants.blogspot.com/2012/10/10-16-12-thoughts-on-bit-packing.html
| OodleXIOQ_ReadMallocWholeFile_Async |
|
|
Discussion
Start a high level IO request to allocate a buffer for a whole file and read itParameters
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
High level IOQ operations are helpers built on the simpler IOQ low level ops.
OodleXIOQ_ReadMallocWholeFile_Async calls OodleXMalloc_IOAligned to allocate a buffer the size of the whole file
(aligned up by OODLEX_IO_MAX_ALIGNMENT), and reads the whole file into that buffer.
Get the buffer pointer with OodleXIOQ_ReadMallocWholeFile_GetResult. You must free it.
Discussion
Log a string describing the OodleXError with OodleXLog_Printf
| OodleX_Init_GetDefaults_Minimal |
|
|
Discussion
Get minimal defaults for OodleXInitOptions , enabling only necessary Oodle systemsParameters
Return Value
Discussion
Fill options such that a minimal part of the Oodle library is imported.
All memory->memory compressors will work.
IO and Threading will be disabled.
Can be used with OodleX_Init_NoThreads or OodleX_Init
| example_lz : Example demonstrating LZ compression and decompression |
|
|
Discussion
Oodle example_lz
Use the various LZ compress/decompress API's
API's demonstrated here :
OodleLZ : low level buffer compress/decompress
OodleLZDecoder : streaming decoder
OodleLZ_Async : high level async helpers
#include "../include/oodle2x.h"
#include "ooex.h"
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include "make_example_input.h"
#ifdef BUILDING_EXAMPLE_CALLER
#define main example_lz
#endif
//===========================================================
// file names :
static const char * in_name = "oodle_example_input_file";
static const char * out_name = "oodle_example_output_file";
// just make the input buffer global here so all the tests can use it :
static void * in_buffer = NULL;
static OO_SINTa in_size = 0;
//===========================================================
// protos :
void lz_test_1();
void lz_test_2();
void lz_test_4();
void lz_test_9();
void lz_test_10();
void lz_test_11();
void lz_test_12();
void lz_test_13();
//===========================================================
extern "C" int main(int argc,char *argv[])
{
OodleXLog_Printf_v1("usage: example_lz [in] [out]\n");
// Init Oodle systems with default options :
OodleXInitOptions opts;
if ( ! OodleX_Init_GetDefaults(OODLE_HEADER_VERSION,&opts) )
{
fprintf(stderr,"Oodle header version mismatch.\n");
return 10;
}
// change opts here if you like
if ( ! OodleX_Init(OODLE_HEADER_VERSION,&opts) )
{
fprintf(stderr,"OodleX_Init failed.\n");
return 10;
}
if ( argc >= 2 ) in_name = argv[1];
else make_example_input(in_name);
if ( argc >= 3 ) out_name = argv[2];
OodleXLog_Printf_v1("lz test %s to %s\n",in_name,out_name);
// read the input file to the global buffer :
OO_S64 in_size_64;
in_buffer = OodleXIOQ_ReadMallocWholeFile_AsyncAndWait(in_name,&in_size_64);
if ( ! in_buffer)
{
OodleXLog_Printf_v0("failed to read %s\n",in_name);
return 10;
}
in_size = OodleX_S64_to_SINTa_check( in_size_64 );
lz_test_1();
lz_test_2();
lz_test_4();
lz_test_9();
lz_test_10();
lz_test_11();
lz_test_12();
lz_test_13();
OodleXLog_Printf_v1("\ndone.\n");
OodleXFree_IOAligned(in_buffer);
//OodleX_Shutdown();
OodleX_Shutdown(NULL,OodleX_Shutdown_LogLeaks_Yes,0);
//OodleXLog_Printf_v1("press a key\n");
//fgetc(stdin);
return 0;
}
//=================================================
lz_test_1 :
example of directly calling the simple buffer->buffer compression API's
OodleLZ_Compress
OodleLZ_Decompress
void lz_test_1()
{
OodleXLog_Printf_v0("lz_test_1\n");
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
void * dec_buf = OodleXMalloc( in_size );
OOEX_ASSERT( dec_buf != NULL );
//---------------------------------------------------
// compress buffer -> buffer :
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_VeryFast;
OO_SINTa comp_len = OodleLZ_Compress(OodleLZ_Compressor_Kraken,in_buffer,in_size,comp_buf,level);
OodleXLog_Printf_v1("Kraken compress %d -> %d\n",(int)in_size,(int)comp_len);
// decompress :
OO_SINTa dec_len = OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( dec_len == in_size );
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
//---------------------------------------------------
// do it again with another compressor, and custom options :
// compress buffer -> buffer :
OodleLZ_CompressOptions options = * OodleLZ_CompressOptions_GetDefault(OodleLZ_Compressor_Leviathan,level);
options.spaceSpeedTradeoffBytes = OODLELZ_SPACESPEEDTRADEOFFBYTES_DEFAULT/2; // favor size over decode speed
comp_len = OodleLZ_Compress(OodleLZ_Compressor_Leviathan,in_buffer,in_size,comp_buf,level,&options);
OodleXLog_Printf_v1("Leviathan compress %d -> %d\n",(int)in_size,(int)comp_len);
// decompress :
dec_len = OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT( dec_len == in_size );
OOEX_ASSERT( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------
// free buffers :
OodleXFree(comp_buf);
OodleXFree(dec_buf);
}
lz_test_2
example of using the OodleLZ_Async_ async helper functions (eg OodleXLZ_Decompress_Wide_Async)
this is the simple way to get the best performance
Use the seekChunkReset option on to get a seekable packed stream.
void lz_test_2()
{
OodleXLog_Printf_v0("lz_test_2\n");
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
void * dec_buf = OodleXMalloc( in_size );
OOEX_ASSERT( dec_buf != NULL );
//---------------------------------------------------
OodleLZ_CompressOptions options = * OodleLZ_CompressOptions_GetDefault(OodleLZ_Compressor_Kraken,OodleLZ_CompressionLevel_Normal);
// turn on seekChunkReset
// this makes chunks independent so they can be decompressed in any order (not just linear)
options.seekChunkReset = true;
options.seekChunkLen = OodleLZ_MakeSeekChunkLen(in_size,8);
// with seekChunkReset on, compression will also go in parallel
// (actually compression can *always* run in parallel, but seekChunkReset makes it scale more linearly,
// and parallelize on a smaller granularity)
// use the OodleLZ_Compressor_Kraken compressor
OodleXHandle h = OodleXLZ_Compress_Async(OodleXAsyncSelect_Full,OodleLZ_Compressor_Kraken,in_buffer,in_size,comp_buf,OodleLZ_CompressionLevel_Normal,&options);
// ... do other game work while compression runs ...
OO_SINTa comp_len = -1;
OodleXLZ_Compress_Wait_GetResult(h,&comp_len);
OodleXLog_Printf_v1("LZ compress %d -> %d\n",(int)in_size,(int)comp_len);
//-----------------------------------------------------
// make seek entries :
// seek entries allow parallel decompression
OodleLZ_SeekTable * seekTable = OodleLZ_CreateSeekTable(OodleLZSeekTable_Flags_None,options.seekChunkLen,in_buffer,in_size,comp_buf,comp_len);
OOEX_ASSERT( seekTable != NULL );
//-----------------------------------------------------
OodleXHandle dh = OodleXLZ_Decompress_Wide_Async(OodleXAsyncSelect_Full,seekTable,comp_buf,comp_len,dec_buf,in_size,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,0,0,OodleLZ_PackedRawOverlap_No,0,0);
// ... do other game work while decompression runs ...
OodleXStatus st = OodleX_Wait(dh,OodleXHandleDeleteIfDone_Yes);
OOEX_ASSERT_ALWAYS( st == OodleXStatus_Done );
//-----------------------------------------------------
// check :
OOEX_ASSERT( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------
// free buffers :
OodleLZ_FreeSeekTable(seekTable);
OodleXFree(comp_buf);
OodleXFree(dec_buf);
}
test_4 :
example of seeking in packed stream
and firing per-chunk decompression tasks
Sort of like what OodleXLZ_Decompress_Wide_Async does internally.
void lz_test_4()
{
OodleXLog_Printf_v0("lz_test_4\n");
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
void * dec_buf = OodleXMalloc( in_size );
OOEX_ASSERT( dec_buf != NULL );
// set up compress options for independent seek chunks of the smallest allowed size :
OodleLZ_CompressOptions lzOptions = * OodleLZ_CompressOptions_GetDefault(OodleLZ_Compressor_Kraken,OodleLZ_CompressionLevel_VeryFast);
lzOptions.seekChunkReset = true;
//lzOptions.seekChunkLen = OODLELZ_BLOCK_LEN;
// make a seek chunk len to target 32 chunks :
lzOptions.seekChunkLen = OodleLZ_MakeSeekChunkLen(in_size,32);
//---------------------------------------------------
// compress buffer -> buffer :
OO_SINTa comp_len = OodleLZ_Compress(OodleLZ_Compressor_Kraken,in_buffer,in_size,comp_buf,OodleLZ_CompressionLevel_VeryFast,&lzOptions);
OodleXLog_Printf_v1("LZ compress %d -> %d\n",(int)in_size,(int)comp_len);
//---------------------------------------------------
// decompress by seeking and firing async decodes
OO_SINTa maxNumSeeks = (in_size + lzOptions.seekChunkLen-1)/lzOptions.seekChunkLen;
OodleXHandle * handles = (OodleXHandle *) OodleXMalloc( sizeof(OodleXHandle) * maxNumSeeks );
OOEX_ASSERT( handles != NULL );
OO_S32 numHandles = 0;
{
OO_SINTa dec_pos = 0;
OO_U8 * comp_ptr = (OO_U8 *)comp_buf;
OO_SINTa comp_avail = comp_len;
while(dec_pos < in_size)
{
OO_SINTa dec_chunk_len = OOEX_MIN(lzOptions.seekChunkLen, (in_size-dec_pos) );
OodleXHandle h = OodleXLZ_Decompress_Narrow_Async(OodleXAsyncSelect_Full,comp_ptr,comp_avail,(OO_U8 *)dec_buf+dec_pos,dec_chunk_len);
// OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None, etc
handles[numHandles++] = h;
OO_SINTa seek_step = OodleLZ_GetCompressedStepForRawStep(comp_ptr,comp_avail,dec_pos,dec_chunk_len);
comp_ptr += seek_step;
dec_pos += dec_chunk_len;
// wait on handles[numHandles-128] to prevent handle count exceeding handle table size :
if ( numHandles >= 128 )
{
OodleXStatus st = OodleX_WaitAndDelete(handles[numHandles-128]);
OOEX_ASSERT_ALWAYS( st == OodleXStatus_Done );
}
}
}
// ... do other game work while async decomps run ...
OodleXStatus st = OodleX_WaitAll(handles,numHandles,OodleXHandleDeleteIfDone_Yes);
OOEX_ASSERT_ALWAYS( st == OodleXStatus_Done );
OodleXFree(handles);
// check it's okay :
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------
// free buffers :
OodleXFree(comp_buf);
OodleXFree(dec_buf);
}
test_9 :
Demonstrate separate block compression & decompression
OodleLZ blocks can be concatenated to form a single valid LZ data stream
That means you can just call OodleLZ_Compress on separate blocks and append the output,
then decode in one call.
OodleLZ blocks that were made from separate Compress calls will be independent
unless you specified dictionary backup in the encode, which makes them depend on previous
data.
The rules are :
1. OodleLZ Decompress can be called on invidual blocks (OODLELZ_BLOCK_LEN) if :
they are seek-chunk-reset points,
OR if they were made by separate OodleLZ Compress calls
OR if the compressor does not carry state across blocks (OodleLZ_Compressor_MustDecodeWithoutResets)
2. OodleLZ Decompress must get the same dictionary as OodleLZ Compress saw
no previous dictionary is needed if it's a seek-chunk-reset point
(the start of an OodleLZ_Compress call is always a seek reset point,
if no dictionary backup is provided to the encoder)
void lz_test_9()
{
OodleXLog_Printf_v0("lz_test_9\n");
//---------------------------------------------------
// split the buffer into two pieces
// such that the split point is a valid seek chunk point :
OO_SINTa block_size = OodleLZ_MakeSeekChunkLen(in_size,2);
if ( block_size >= in_size )
{
// too small to split at seek chunk
return;
}
char * in1 = (char *)in_buffer;
char * in2 = in1 + block_size;
OO_SINTa len1 = block_size;
OO_SINTa len2 = in_size - len1;
OodleXLog_Printf_v1("Chunks : %d + %d\n",(int)len1,(int)len2);
//---------------------------------------------------
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
void * dec_buf = OodleXMalloc( in_size );
OOEX_ASSERT( dec_buf != NULL );
//---------------------------------------------------
OodleLZ_Compressor compressor = OodleLZ_Compressor_Kraken;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Fast;
OodleLZ_CompressOptions options = * OodleLZ_CompressOptions_GetDefault(compressor,level);
// options does NOT have seek resets by default
//-----------------------------
// compress as one part :
{
OO_SINTa comp_len = OodleLZ_Compress(compressor,in_buffer,in_size,comp_buf,level,&options);
OodleXLog_Printf_v1("Whole buffer compress : %d -> %d\n",(int)in_size,(int)comp_len);
// normal one part decompression :
memset(dec_buf,0xEE,in_size);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
OO_U8 * comp_end = (OO_U8 *)comp_buf + comp_len;
//===================================================================
// can decode in two calls with the full dictionary, but only for compressors that don't carry state :
if ( ! OodleLZ_Compressor_MustDecodeWithoutResets(compressor) )
{
//-------------------------------------------------------------
// decode as two parts : (len1,len2) :
memset(dec_buf,0xEE,in_size);
OO_SINTa dec_comp_len1 = OodleLZ_GetCompressedStepForRawStep(comp_buf,comp_len,0,len1);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,len1,OodleLZ_FuzzSafe_Yes);
// decompress second part with dictionary base :
OodleLZ_Decompress((char *)comp_buf+dec_comp_len1,comp_len-dec_comp_len1,(char *)dec_buf+len1,len2,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,dec_buf,in_size);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------------------------------
// can also decode block by block :
memset(dec_buf,0xEE,in_size);
// scan comp_ptr through blocks :
OO_U8 * comp_ptr = (OO_U8 *)comp_buf;
for(OO_SINTa block_pos=0;block_pos < in_size;block_pos += OODLELZ_BLOCK_LEN)
{
OO_SINTa block_len = OOEX_MIN(OODLELZ_BLOCK_LEN, (in_size - block_pos) );
OO_SINTa block_comp_len = OodleLZ_GetCompressedStepForRawStep(comp_ptr,comp_end-comp_ptr,0,block_len);
// decode current block, with window set to whole buffer :
OO_SINTa got_pos = OodleLZ_Decompress(comp_ptr,block_comp_len,(char *)dec_buf+block_pos,block_len,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,dec_buf,in_size);
OOEX_ASSERT_ALWAYS( got_pos == block_pos+block_len );
comp_ptr += block_comp_len;
}
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------------------------------
}
}
//-----------------------------
// two part compression with overlap :
// two compress calls, but using the full window, so decompression must use full window as well
{
OO_SINTa comp_len1 = OodleLZ_Compress(compressor,in1,len1,comp_buf,level,&options,in_buffer);
OO_SINTa comp_len2 = OodleLZ_Compress(compressor,in2,len2,(char *)comp_buf + comp_len1,level,&options,in_buffer);
OO_SINTa comp_len = comp_len1 + comp_len2;
OodleXLog_Printf_v1("Two part compress with overlap : %d -> %d\n",(int)in_size,(int)comp_len);
// must decode whole buffer, but can do it in two calls :
// you can always just do a whole buffer decode here :
memset(dec_buf,0xEE,in_size);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
// or incremental, but with the whole dictionary :
memset(dec_buf,0xEE,in_size);
OO_SINTa dec_comp_len1 = OodleLZ_GetCompressedStepForRawStep(comp_buf,comp_len,0,len1);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,len1,OodleLZ_FuzzSafe_Yes);
OodleLZ_Decompress((char *)comp_buf+dec_comp_len1,comp_len-dec_comp_len1,(char *)dec_buf+len1,len2,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,dec_buf,in_size);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
}
//-----------------------------
// two part no overlap :
// second compress doesn't use earlier dictionary here
// so decompression can be done in two pieces with no overlap
{
OO_SINTa comp_len1 = OodleLZ_Compress(compressor,in1,len1,comp_buf,level,&options);
OO_SINTa comp_len2 = OodleLZ_Compress(compressor,in2,len2,(char *)comp_buf + comp_len1,level,&options);
OO_SINTa comp_len = comp_len1 + comp_len2;
OodleXLog_Printf_v1("Two part compress no overlap : %d -> %d\n",(int)in_size,(int)comp_len);
// can decode in two parts :
// you can always just do a whole buffer decode here :
memset(dec_buf,0xEE,in_size);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
// or incremental :
memset(dec_buf,0xEE,in_size);
OO_SINTa dec_comp_len1 = OodleLZ_GetCompressedStepForRawStep(comp_buf,comp_len,0,len1);
// no dictionary backup needed
// decode in reverse order to simulate random access :
OodleLZ_Decompress((char *)comp_buf+dec_comp_len1,comp_len-dec_comp_len1,(char *)dec_buf+len1,len2,OodleLZ_FuzzSafe_Yes);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,len1,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
}
//-----------------------------
// two part no overlap via seek reset :
// seek reset system is equivalent to splitting Compress calls like the above
{
options.seekChunkReset = true;
options.seekChunkLen = (OO_S32)block_size;
OodleXLog_Printf_v1("seekChunkLen : %d\n",options.seekChunkLen);
OO_SINTa comp_len = OodleLZ_Compress(compressor,in_buffer,in_size,comp_buf,level,&options);
OodleXLog_Printf_v1("Whole buffer compress seek reset : %d -> %d\n",(int)in_size,(int)comp_len);
// can decode in two parts :
// you can always just do a whole buffer decode here :
memset(dec_buf,0xEE,in_size);
OodleXLog_Printf_v1("one part : \n");
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
// or incremental :
memset(dec_buf,0xEE,in_size);
OodleXLog_Printf_v1("two part : \n");
OO_SINTa dec_comp_len1 = OodleLZ_GetCompressedStepForRawStep(comp_buf,comp_len,0,len1);
OodleXLog_Printf_v1("dec_comp_len1 = %d\n",(int)dec_comp_len1);
// no dictionary backup needed
// decode in reverse order to simulate random access :
OodleLZ_Decompress((char *)comp_buf+dec_comp_len1,comp_len-dec_comp_len1,(char *)dec_buf+len1,len2,OodleLZ_FuzzSafe_Yes);
OodleLZ_Decompress(comp_buf,comp_len,dec_buf,len1,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
}
//=============================================
OodleXFree(dec_buf);
OodleXFree(comp_buf);
}
lz_test_10 :
example of using the incremental/streaming decoder
OodleLZDecoder_Create, etc.
this example shows decoding *from* a limited window
outputs into a single buffer
this example simulates using a limited IO buffer for compressed data
it decodes quanta from the available compressed data
void lz_test_10()
{
OodleXLog_Printf_v0("lz_test_10\n");
// allocate compressed buffer & decoded buffer of the correct sizes :
OodleLZ_Compressor compressor = OodleLZ_Compressor_Kraken;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Fast;
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
OO_U8 * comp_buf = (OO_U8 *) OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
OO_U8 * dec_buf = (OO_U8 *)OodleXMalloc( in_size );
OOEX_ASSERT( dec_buf != NULL );
//---------------------------------------------------
// compress buffer -> buffer :
OO_SINTa comp_len = OodleLZ_Compress(compressor,in_buffer,in_size,comp_buf,level);
OodleXLog_Printf_v1("LZ compress %d -> %d\n",(int)in_size,(int)comp_len);
//---------------------------------------------------
// decompress with incremental streaming decoder :
// we're now going to pretend that "comp_buf" is in a file
// and we can't read the whole thing
// 64k IO buffer to stress the code :
// obviously you would use much larger
// must be at least enough for 1 whole compressed quantum (OODLELZ_BLOCK_MAX_COMPLEN = 256k + 2)
// (for Kraken / whole-block compressors you probably want 512k io_buffer minimum)
const OO_SINTa io_buffer_size = OODLELZ_BLOCK_LEN*2;
OO_U8 * io_buffer = (OO_U8 *) OodleXMalloc_IOAligned(io_buffer_size);
OOEX_ASSERT( io_buffer != NULL );
OO_SINTa io_buffer_avail = 0; // starts empty
OO_SINTa comp_file_io_pos = 0; // simulated compressed file next read pos
{
// make the Decoder object using on-stack memory :
//OO_S32 memSize = OodleLZDecoder_MemorySizeNeeded(compressor,in_size);
OodleLZDecoder * decoder = OodleLZDecoder_Create(compressor,in_size,NULL,0);
OO_SINTa io_buffer_pos = 0;
OO_U8 * dec_buf_ptr = (OO_U8 *) dec_buf;
OO_U8 * dec_buf_end = (OO_U8 *) dec_buf + in_size;;
while(dec_buf_ptr<dec_buf_end)
{
// see if we can do a "read" into the io_buffer :
if ( comp_file_io_pos < comp_len )
{
// don't bother with an IO unless I have some minimum amount of room :
OO_SINTa min_io_size = 16*1024;
if ( (io_buffer_size - io_buffer_avail) > min_io_size )
{
OO_SINTa io_size = OOEX_MIN( (io_buffer_size - io_buffer_avail), (comp_len - comp_file_io_pos) );
// stress - limit IO size :
//io_size = OOEX_MIN(io_size,64*1024);
//OodleXLog_Printf_v1("IO read : %d at %d\n",(int)io_size,(int)comp_file_io_pos);
// IO read :
memcpy(io_buffer+io_buffer_avail,comp_buf+comp_file_io_pos,io_size);
comp_file_io_pos += io_size;
io_buffer_avail += io_size;
}
}
// ask the Decoder for a partial decode :
OodleLZ_DecodeSome_Out out;
OO_BOOL ok = OodleLZDecoder_DecodeSome(decoder,&out,
dec_buf,(dec_buf_ptr - dec_buf),in_size,(dec_buf_end - dec_buf_ptr),
io_buffer+io_buffer_pos,io_buffer_avail-io_buffer_pos);
// real usage should check error return conditions
OOEX_ASSERT_ALWAYS( ok );
OO_S32 decoded = out.decodedCount;
OO_S32 comp_used = out.compBufUsed;
// advance the decoder :
dec_buf_ptr += decoded;
io_buffer_pos += comp_used;
OOEX_ASSERT( out.curQuantumCompLen < io_buffer_size );
//OodleXLog_Printf_v1("decoded : %d using %d\n",decoded,comp_used);
if ( decoded == 0 )
{
// couldn't decode anything
// this should only happen because we're near the end of the io buffer
// and don't have enough compressed data to do antyhing
OOEX_ASSERT( io_buffer_pos > 0 );
// slide down the io buffer so it can refill
OO_SINTa io_buffer_keep = io_buffer_avail - io_buffer_pos;
memmove(io_buffer,io_buffer+io_buffer_pos,io_buffer_keep);
io_buffer_pos = 0;
io_buffer_avail = io_buffer_keep;
}
}
OOEX_ASSERT( comp_file_io_pos == comp_len );
OodleLZDecoder_Destroy(decoder);
}
// check it's okay :
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_buf,in_size) == 0 );
//-------------------------------------
// free buffers :
OodleXFree(comp_buf);
OodleXFree(dec_buf);
OodleXFree_IOAligned(io_buffer);
}
lz_test_11 :
example of using the incremental/streaming decoder
OodleLZDecoder_Create, etc.
this example simulates using a limited IO buffer for compressed data (like lz_test_10)
Kraken does not currently have a true "sliding window" decoder; it can't wrap around a circular
window. This example shows how to simulate a sliding window with the Kraken decoder by sliding down chunks.
It decodes 256k at a time into a 512k window. It decodes into the second half of the window, with
the first window filled by the previous decode. After each decode, it memcopies down the data to be used
as dictionary for the next block.
---------
In general this method should not be used if you can just decode directly into the output buffer.
That's always the best way if possible.
One case where you might want to use this is if your output buffer is in non-cached graphics memory.
---------
The simpler alternative to this is just to reset every 256k block, so there's no dictionary overlap.
eg. just set :
options.seekChunkReset = true;
options.seekChunkLen = OODLELZ_BLOCK_LEN;
then you can use a 256k decode output window and don't need to memcpy to slide down the dictionary.
The disadvantage of resetting is just lower compression.
/
// slowest encoder:
OodleLZ_Compressor compressor = OodleLZ_Compressor_Leviathan;
OodleLZ_CompressionLevel level = OodleLZ_CompressionLevel_Optimal5;
/*
//---------------------------------------------------
// minimum size :
// we will decode 256k (one "block") at a time
// +256k more for dictionary references to preceding data
OO_S32 decode_window_size = 2*OODLELZ_BLOCK_LEN; // OODLELZ_BLOCK_LEN = 256k
OO_S32 dictionary_size = OODLELZ_BLOCK_LEN;
/
// more reasonable size :
// dictionary limit 2M
// decode in a 3M window, so we do a memcpy after every 1M streamed
OO_S32 decode_window_size = 3*1024*1024; // OODLELZ_BLOCK_LEN = 256k
OO_S32 dictionary_size = 2*1024*1024;
OodleXLog_Printf_v1("dictionary_size : %d, decode_window_size : %d\n",dictionary_size,decode_window_size);
//---------------------------------------------------
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(compressor,in_size);
OO_U8 * comp_buf = (OO_U8 *) OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
// dec_window is our scratch circular window
OO_U8 * dec_window = (OO_U8 *)OodleXMalloc( decode_window_size );
OOEX_ASSERT( dec_window != NULL );
// dec_out_buf is the final output location (perhaps uncached graphics memory)
OO_U8 * dec_out_buf = (OO_U8 *)OodleXMalloc( in_size );
OOEX_ASSERT( dec_out_buf != NULL );
// decoderMem is used for the OodleLZ decoder object
OO_S32 memSize = OodleLZDecoder_MemorySizeNeeded(compressor);
OO_U8 * decoderMem = (OO_U8 *) OodleXMalloc(memSize);
//---------------------------------------------------
// compress buffer -> buffer :
// limit dictionarySize so matches can't go out of the decode window :
OodleLZ_CompressOptions options = * OodleLZ_CompressOptions_GetDefault(compressor,level);
options.dictionarySize = dictionary_size;
OO_SINTa comp_len = OodleLZ_Compress(compressor,in_buffer,in_size,comp_buf,level,&options);
OodleXLog_Printf_v1("LZ compress %d -> %d\n",(int)in_size,(int)comp_len);
//---------------------------------------------------
// decompress with incremental streaming decoder :
// we're now going to pretend that "comp_buf" is in a file
// and we can't read the whole thing
// IO buffer must be at least enough for 1 whole quantum (256k + a little bit)
// Kraken uses "large block quantum" (256k) not the old 16k quantum
const OO_SINTa io_buffer_size = (256+63)*1024;
OO_U8 io_buffer[io_buffer_size];
OO_SINTa io_buffer_avail = 0; // starts empty
OO_SINTa comp_file_io_pos = 0; // simulated compressed file next read pos
{
OodleLZDecoder * decoder = OodleLZDecoder_Create(compressor,in_size,decoderMem,memSize);
OO_SINTa io_buffer_pos = 0;
OO_U8 * dec_out_ptr = (OO_U8 *) dec_out_buf;
OO_U8 * dec_out_end = (OO_U8 *) dec_out_buf + in_size;;
OO_SINTa dec_window_pos = 0;
while(dec_out_ptr<dec_out_end)
{
// see if we can do a "read" into the io_buffer :
if ( comp_file_io_pos < comp_len )
{
OO_SINTa min_io_size = 16*1024;
if ( (io_buffer_size - io_buffer_avail) > min_io_size )
{
OO_SINTa io_size = OOEX_MIN( (io_buffer_size - io_buffer_avail), (comp_len - comp_file_io_pos) );
//OodleXLog_Printf_v1("IO read : %d at %d\n",(int)io_size,(int)comp_file_io_pos);
// IO read :
memcpy(io_buffer+io_buffer_avail,comp_buf+comp_file_io_pos,io_size);
comp_file_io_pos += io_size;
io_buffer_avail += io_size;
}
}
// when dec_window_pos reaches end of window :
if ( (dec_window_pos + OODLELZ_BLOCK_LEN) > decode_window_size )
{
OodleXLog_Printf_v1("slide!\n");
// slide down the dictionary for the next block :
memmove(dec_window,dec_window + dec_window_pos - dictionary_size,dictionary_size);
dec_window_pos = dictionary_size;
}
OodleXLog_Printf_v1("decode : at %d in window, %d in output\n",(int)dec_window_pos,(int)(dec_out_ptr - dec_out_buf));
// ask the Decoder for a partial decode :
OodleLZ_DecodeSome_Out out;
// no need to truncate dec_avail at the end, the "in_size" passed to LZDecoder_Create does this
//OO_SINTa dec_avail = OOEX_MIN(dec_out_remain,decode_window_size - dec_window_pos);
OO_SINTa dec_avail = decode_window_size - dec_window_pos;
OO_BOOL ok = OodleLZDecoder_DecodeSome(decoder,&out,
dec_window,dec_window_pos,
in_size, //decode_window_size, // !!
dec_avail,
io_buffer+io_buffer_pos,io_buffer_avail-io_buffer_pos,
OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No);
// !! = this is a bit funny; we lie about the window size here
// if Kraken had a true sliding window decoder (like eg. LZH or LZB16 does)
// then you would pass decode_window_size here and let it do the wrapping
// but Kraken does not, so we pretend that we are decoding the whole file
// so that OodleLZDecoder_DecodeSome doesn't try to use its sliding window path
// (which would fail)
// the "dec_avail" value prevents DecodeSome from going out of the window
// and we simulate the sliding using the memcpy
// real usage should check error return conditions
OOEX_ASSERT_ALWAYS( ok );
OO_S32 decoded = out.decodedCount;
OO_S32 comp_used = out.compBufUsed;
io_buffer_pos += comp_used;
OOEX_ASSERT( out.curQuantumCompLen < io_buffer_size );
//OodleXLog_Printf_v1("decoded : %d using %d\n",decoded,comp_used);
if ( decoded == 0 )
{
// couldn't decode anything
// this should only happen because we're near the end of the io buffer
// and don't have enough compressed data to do antyhing
OOEX_ASSERT( io_buffer_pos > 0 );
// slide down the io buffer so it can refill
OO_SINTa io_buffer_keep = io_buffer_avail - io_buffer_pos;
memmove(io_buffer,io_buffer+io_buffer_pos,io_buffer_keep);
io_buffer_pos = 0;
io_buffer_avail = io_buffer_keep;
}
else
{
// copy out the decoded data :
OO_U8 * dec_window_ptr = dec_window + dec_window_pos;
// dec_out_ptr is the final output memory ; eg. perhaps uncached graphics memory
// "decoded" is always OODLELZ_BLOCK_LEN unless we hit EOF
memcpy(dec_out_ptr,dec_window_ptr,decoded);
// advance the decoder :
dec_out_ptr += decoded;
dec_window_pos += decoded;
}
}
OOEX_ASSERT( comp_file_io_pos == comp_len );
OOEX_ASSERT( dec_out_ptr == dec_out_end );
OodleLZDecoder_Destroy(decoder);
}
// check it's okay :
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,dec_out_buf,in_size) == 0 );
//-------------------------------------
// free buffers :
OodleXFree(decoderMem);
OodleXFree(comp_buf);
OodleXFree(dec_out_buf);
OodleXFree(dec_window);
}
lz_test_12 :
example of directly calling the simple buffer->buffer compression API's using an "in place" buffer
OodleLZ_Compress
OodleLZ_Decompress
OodleLZ_GetInPlaceDecodeBufferSize
void lz_test_12()
{
OodleXLog_Printf_v0("lz_test_12\n");
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken,in_size);
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
//---------------------------------------------------
// compress buffer -> buffer :
OO_SINTa comp_len;
{
comp_len = OodleLZ_Compress(OodleLZ_Compressor_Kraken,in_buffer,in_size,comp_buf,OodleLZ_CompressionLevel_Fast);
}
//---------------------------------------------------
OO_SINTa inplace_size = OodleLZ_GetInPlaceDecodeBufferSize(OodleLZ_Compressor_Kraken,comp_len,in_size);
OodleXLog_Printf_v1("Kraken compress %d -> %d ; inplace_size = %d , padding = %d\n",
(int)in_size,(int)comp_len,
(int)inplace_size,(int)(inplace_size - in_size));
void * inplace_buf = OodleXMalloc( inplace_size );
OOEX_ASSERT( inplace_buf != NULL );
// in game use, you load the compressed data into the *end* of the inplace buffer
// simulate the loading by doing a memcpy :
char * inplace_comp_ptr = (char *)inplace_buf;
inplace_comp_ptr += inplace_size - comp_len;
memcpy(inplace_comp_ptr,comp_buf,comp_len);
//---------------------------------------------------
// decompress :
// note the source (inplace_comp_ptr) and dest (inplace_buf) overlap
// - the compressed data at inplace_comp_ptr is destroyed by this call
OO_SINTa dec_len = OodleLZ_Decompress(inplace_comp_ptr,comp_len,inplace_buf,in_size,OodleLZ_FuzzSafe_Yes);
OOEX_ASSERT_ALWAYS( dec_len == in_size );
OOEX_ASSERT_ALWAYS( memcmp(in_buffer,inplace_buf,in_size) == 0 );
//---------------------------------------------------
// free buffers :
OodleXFree(comp_buf);
OodleXFree(inplace_buf);
}
//=================================================
lz_test_13 :
example of dictionary-relative compression
This is a technique in which you train a dictionary offline based on typical data, then for
each packet you wish to compress, the dictionary can be used a reference to improve compression ratio.
Oodle can do dictionary relative compression by putting the packet buffer to compress
in a contiguous buffer immediately following the dictionary.
Then simply use memcpy to move the active packet to the desired memory location.
NOTE that the work space for {dictionary + packet} must be allocated per thread, or mutex controlled
(it cannot be shared by simultaneously decoding threads)
For small packets (under 4 KB or so) such as network packets, consider Oodle Network instead.
For large buffers (over 128 KB or so), dictionary-relative compression doesn't help much and isn't recommended.
dictionary-relative compression is most typically useful on data in the 4 - 128 KB range.
void lz_test_13()
{
OodleXLog_Printf_v0("lz_test_13\n");
//---------------------------------------------------
// pretend that "in_buffer" consists of a trained dictionary + a packet to compress
void * dictionary = in_buffer;
OO_SINTa dictionary_size = (in_size * 2) /3;
// dictionary_size must be a multiple of OODLELZ_BLOCK_LEN :
dictionary_size &= ~(OODLELZ_BLOCK_LEN-1);
void * packet1 = (char *)dictionary + dictionary_size;
OO_SINTa packet1_size = in_size / 4;
void * packet2 = (char *)packet1 + packet1_size;
OO_SINTa packet2_size = in_size - packet1_size - dictionary_size;
OodleXLog_Printf_v1("dictionary_size : %d ; packets : %d + %d\n",dictionary_size,packet1_size,packet2_size);
//---------------------------------------------------
// allocate compressed buffer & decoded buffer of the correct sizes :
OO_SINTa max_packet_size = OOEX_MAX(packet1_size,packet2_size);
// room for dictionary + a packet following :
void * dictionary_and_packet_buf = OodleXMalloc( dictionary_size + max_packet_size );
OOEX_ASSERT( dictionary_and_packet_buf != NULL );
// comp buf just for a packet :
OO_SINTa comp_buf_size = OodleLZ_GetCompressedBufferSizeNeeded(OodleLZ_Compressor_Kraken, max_packet_size );
void * comp_buf = OodleXMalloc( comp_buf_size );
OOEX_ASSERT( comp_buf != NULL );
// room for dictionary + a packet following :
void * dec_buf = OodleXMalloc( dictionary_size + max_packet_size );
OOEX_ASSERT( dec_buf != NULL );
//---------------------------------------------------
void * packets[2] = { packet1, packet2 };
OO_SINTa packet_sizes[2] = { packet1_size, packet2_size };
// setup work that's done in advance :
// put dictionary at head of dictionary_and_packet_buf (for encoder) :
memcpy(dictionary_and_packet_buf,dictionary,dictionary_size);
void * after_dictionary_ptr = (char *)dictionary_and_packet_buf + dictionary_size;
// preload dictionary at head of dec_buf (for decoder) :
memcpy(dec_buf,dictionary,dictionary_size);
for(int packet_i=0;packet_i<2;packet_i++)
{
// work that's done per packet :
void * packet_ptr = packets[packet_i];
OO_SINTa packet_size = packet_sizes[packet_i];
// compress packet to comp_buf
// preload with dictionary
// copy packet to be immediately following dictionary :
memcpy(after_dictionary_ptr,packet_ptr,packet_size);
OO_SINTa comp_len = OodleLZ_Compress(OodleLZ_Compressor_Kraken,after_dictionary_ptr,packet_size,comp_buf,OodleLZ_CompressionLevel_Fast,NULL,dictionary_and_packet_buf);
OodleXLog_Printf_v1("Kraken compress %d -> %d\n",(int)packet_size,(int)comp_len);
// decompress :
// decode into buffer containing dictionary, immediately following dictionary :
void * dec_packet_ptr = (char *)dec_buf + dictionary_size;
OO_SINTa dec_len = OodleLZ_Decompress(comp_buf,comp_len,dec_packet_ptr,packet_size,OodleLZ_FuzzSafe_Yes,OodleLZ_CheckCRC_No,OodleLZ_Verbosity_None,dec_buf);
OOEX_ASSERT_ALWAYS( dec_len == packet_size + dictionary_size );
OOEX_ASSERT_ALWAYS( memcmp(packet_ptr,dec_packet_ptr,packet_size) == 0 );
// if you need the decoded packet to be in another memory location, memcpy it there now
}
//---------------------------------------------------
// free buffers :
OodleXFree(dictionary_and_packet_buf);
OodleXFree(comp_buf);
OodleXFree(dec_buf);
}
Oodle on WASM is currently provided as only the Oodle Core lib (no OodleX).
This includes the synchronous LZ compressors. For WASM, the library tries to have zero
runtime dependencies. Among other things, that means that unlike other Oodle platforms,
it does not have a default memory allocator or logging implementation.
The Oodle compressors and decompressors do not require dynamic allocations as long as
sufficient "scratch"/"decoder" memory is passed in; refer to the documentation for
OodleLZ_Compress and OodleLZ_Decompress for details. You need to either do this or
install a memory allocator via OodleCore_Plugins_SetAllocators. If you do neither,
Oodle compression/decompression requests will fail.
Likewise, if you want to see Oodle diagnostic log messages, install a printf plugin via
OodleCore_Plugins_SetPrintf.
Finally, Oodle on WASM does not include the Optimal level encoders. When OodleLZ_CompressionLevel_Optimal1 or
higher is requested, the encoder falls back to a variant of OodleLZ_CompressionLevel_Normal instead.
struct OodleLZ_DecodeSome_Out
{
OO_S32 decodedCount;
OO_S32 compBufUsed;
OO_S32 curQuantumRawLen;
OO_S32 curQuantumCompLen;
};
Discussion
Output value of OodleLZDecoder_DecodeSome
Members
| decodedCount | number of uncompressed bytes decoded |
| compBufUsed | number of compressed bytes consumed |
| curQuantumRawLen | tells you the current quantum size. you must have at least this much room available in the output buffer to be able to decode anything. |
| curQuantumCompLen | if you didn't pass in enough data, nothing will decode (decodedCount will be 0), and this will tell you how much is needed |
Discussion
Convert an OS error code into a text messageParameters
Return Value
Discussion
Converts an OS-specific error code into a platform agnostic error enum. Useful for
recognizing common error cases like OodleXError_FileNotFound. Any unusual or platform-specific
codes will return OodleXError_Unknown.
| FAQ: My Files aren't loading right and I can't track it down |
|
|
It can be hard to diagnose IO problems when using async IO, because the actual error is removed from
the call that caused the error.
Here are some steps that may help :
1. Make your own OodleXInitOptions struct to pass to OodleX_Init so we can
change some of the flags. Use OodleX_Init_GetDefaults to fill it out at first.
The members OodleXInitOptions:m_OodleInit_Log and OodleXInitOptions:m_OodleInit_ThreadLog
should be on by default, don't turn them off.
2. Set OodleXInitOptions:m_OodleInit_IOQ_Log to true ; this will enable all IO
to be logged, and you may see what's causing the error.
3. Set OodleXInitOptions:m_OodleInit_IOQ_Threaded to false ; this will disable
IO threading so the error will occur at the call site.
4. Set OodleXInitOptions:m_OodleInit_IOQ_BreakOnError to true; this will cause
Oodle IO to do a debug break when it encounters an error. If using this, make sure
you are linking with the debug build of Oodle, and using the debug dll on Windows
(in the "redistdebug" directory). You should now be able to see the stack of
the call causing the error, and your log should contain information about the
failure.
5. If your app is crashing because of the error and you can't debug, you
can set OodleXInitOptions:m_OodleInit_Log_FlushEachWrite ; hopefully this will
get the log to flush out some information about the error before the crash.
The log file is written to c:\oodlelogs on Windows unless you have set m_OodleInit_Log_FileName.
(see About Oodle on Platforms for per-platform log info, or set the log name yourself in OodleXInitOptions)
| OodleNet_Plugins_SetAllocators |
|
|
Discussion
Set the function pointers for allocation needed by Oodle2 CoreDiscussion
If these are not set, the default implementation on most platforms uses the C stdlib.
On Microsoft platforms the default implementation uses HeapAlloc.
These must not be changed once they are set! Set them once then don't change them.
If you want to ensure that Oodle is not doing any allocations, you can call OodleNet_Plugins_SetAllocators(NULL,NULL);
If you do that, then any time Oodle needs to allocate memory internally, it will stop the process.
It is STRONGLY not recommended that you ship that way. You can verify that Oodle is not allocating, but then leave some
fallback allocator installed when you actually ship just in case.
Also note that on many consoles the standard allocation practices may not
leave much heap memory for the C stdlib malloc. In this case Oodle may fail to allocate.
| OodleNet_Plugins_SetJobSystemAndCount |
|
|
Discussion
Set the function pointers for async job system needed by Oodle2 CoreParameters
Discussion
If these are not set, the default implementation runs jobs synchronously on the calling thread.
These must not be changed once they are set! Set them once then don't change them.
target_parallelism allows you to tell Oodle how many Jobs it should try to keep in flight at once.
Depending on the operation it may not be able to split work into this many jobs (so fewer will be used),
but it will not exceed this count.
For Oodle Data LZ work, typically target_parallelism is usually best at the number of hardware cores
not including hyper threads).
For Oodle Texture BCN encoding work, target_parallelism is usually best as the full number of hyper cores.
In some cases you may wish to reduce target_parallelism by 1 or 2 cores to leave some of the CPU free for
other work.
For example on a CPU with 16 cores and 32 hardware threads, for LZ work you might set target_parallelism to 15
when calling OodleCorePlugins. For BC7 encoding you might set target_parallelism to 30 when calling OodleTexPlugins.
NOTE : if you are using Oodle Ext, do NOT call this. OodleX_Init will install a job system for Oodle Core.
Note OodleX only installs automatically to Oodle Core, not Net or Tex. See example_jobify.cpp for manual
plugin.
Replaces deprecated OodleNet_Plugins_SetJobSystem
See About Oodle Job Threading Plugins
struct OodleXConfigValues
{
OO_S32 m_Oodle_DefaultIOBufferSize;
OO_S32 m_Oodle_DefaultWriteReserveSize;
OO_S32 m_Oodle_MaxSingleIOSize;
OO_S32 m_OodleIOQStream_MaxReadSize;
OO_S32 m_OodleIOQStream_MinReadSize;
OO_S32 m_OodleIOQStream_OffsetAlignment;
OO_S32 m_Oodle_very_long_wait_seconds;
OO_S32 m_deprecated_Desired_Parallel_BranchFactor;
OO_BOOL m_Oodle_OSFileOpen_Default_Read_Buffered;
OO_BOOL m_Oodle_OSFileOpen_Default_Write_Buffered;
OO_BOOL m_Oodle_PathsCaseSensitive;
OO_U32 m_oodle_header_version;
};
Discussion
OodleXConfigValuesMembers
Discussion
Struct of user-settable low level config values. See OodleX_SetConfigValues.
May have different defaults per platform.
Discussion
Set the VTable used for ops the file.Parameters
| file | the IOQFile to query
|
Return Value
Discussion
Change the VTable used for ops the file after opening. This is discouraged, generally try to set the right
vtable in the OodleXIOQ_OpenForRead_Async call and then don't change it.
Warning : vtables are not themselves internally mutex protected !
WARNING : changing the file's VTable while there are ops on that file in the Queue has undefined results !!
| FAQ: Which OodleLZ should I use? |
|
|
The primary Oodle compressors you might want to use are : Kraken, Mermaid, Selkie, and Leviathan.
They are designed for slightly different usage scenarios. This is a simple guide to choosing one.
(See also About OodleLZ for more detailed information about OodleLZ)
The best way to sample them is just to run example_lz_chart : Example that makes a chart of OodleLZ options on some of your data.
All the Oodle compressors support seeking in packed streams and parallel decompression, if you set up
the OodleLZ_CompressOptions to request that. (set seekChunkReset)
Kraken is a great general purpose place to start. It has high compression and great decode speed.
Mermaid & Selkie are some of the fastest decompressors in the world. They provide less compression than Kraken but
super fast decodes. Selkie is faster than LZ4.
Leviathan offers the highest compression. It's comparable to LZMA (7zip), but a lot faster to decode!
Again :
- Kraken should be your default choice. For good compression and good decode speed, use Kraken.
- For maximum decompression speed, use Selkie. For very fast decoding with more compression, use Mermaid.
(Mermaid is between Selkie and Kraken)
- Kraken, Mermaid, Selkie and Leviathan should normally be your first choice,
but they do better on larger files and may not do well on very small files.
- Unless you know that you specifically need a small sliding window or seeking, you should
compress buffers in the largest pieces possible. In general, cutting data into pieces
(via seekChunkReset or your own cutting) reduces compression ratio.
See also FAQ: What are the speeds and ratios of the OodleLZ compressors and levels?.
Discussion
Opaque data type for OodleNetwork1_SharedDiscussion
This data is made from the shared static dictionary. After it is made it is const,
and can be used by all compression channels.
This data is filled by OodleNetwork1_Shared_SetWindow.
You can allocate and free it yourself. It must be of size OodleNetwork1_Shared_Size.
Your server must have one of these, and each client must have the exact same one.
| OodleNetwork1_CompressedBufferSizeNeeded |
|
|
Discussion
Returns the size of memory required for the compressed buffer passed to OodleNetwork1TCP_Encode
Discussion
Get the OS file handle for this OodleXIOQFileParameters
| file | the IOQFile to query
|
Return Value
Discussion
If the file is not yet open (eg. OodleXIOQ_OpenForRead_Async was started but is still pending), this
returns NULL.
Discussion
Unknown or failure retreiving file size
| OodleNetwork1UDP_State_Compact |
|
|
Discussion
Fills a OodleNetwork1UDP_StateCompacted from a OodleNetwork1UDP_StateParameters
Return Value
| return | number of bytes filled in to
|
Discussion
Use this when the OodleNetwork1UDP_State is created to make a Compacted state to save to a file.
to should be allocated to at least OodleNetwork1UDP_StateCompacted_MaxSize
Note - use the return value to save only the prefix of the Compacted state.
enum OodleXLog_StateFlags
{
OODLEXLOG_TO_FILE = ((OO_U32)1<<0),
OODLEXLOG_ECHO = ((OO_U32)1<<1),
OODLEXLOG_TO_DEBUGGER = ((OO_U32)1<<2),
OODLEXLOG_FILE_LINE = ((OO_U32)1<<3),
OODLEXLOG_CALLBACK = ((OO_U32)1<<4),
OODLEXLOG_PREFIX_THREAD_TIME = ((OO_U32)1<<5),
OODLEXLOG_AUTOFLUSH_THREADLOG = ((OO_U32)1<<6),
OODLEXLOG_FLUSH_EVERY_WRITE = ((OO_U32)1<<7),
OODLEXLOG_STATE_VERBOSITY_NONE = ((OO_U32)0<<16),
OODLEXLOG_STATE_VERBOSITY0 = ((OO_U32)1<<16),
OODLEXLOG_STATE_VERBOSITY1 = ((OO_U32)2<<16),
OODLEXLOG_STATE_VERBOSITY2 = ((OO_U32)3<<16)
};
Discussion
Flags for use with OodleXLog_SetState
Enumerants
enum OodleLZ_Verbosity
{
OodleLZ_Verbosity_None = 0,
OodleLZ_Verbosity_Minimal = 1,
OodleLZ_Verbosity_Some = 2,
OodleLZ_Verbosity_Lots = 3,
OodleLZ_Verbosity_Force32 = 0x40000000
};
Discussion
Verbosity of LZ functions
LZ functions print information to the function set by OodleCore_Plugins_SetPrintf
or OodleXLog_Printf if using OodleX.
| t_fp_OodleNet_Plugin_WaitJob |
|
|
Discussion
Function pointer type for OodleNet_Plugins_SetJobSystemParameters
| job_handle | a job handle returned from RunJob. Never 0.
|
| user_ptr | is passed through from the OodleLZ_CompressOptions.
|
Discussion
Waits until the job specified by job_handle is done and cleans up any associated resources. Oodle
will call WaitJob exactly once for every RunJob call that didn't return 0.
If job_handle was already completed, this should clean it up without waiting.
A handle value should not be reused by another RunJob until WaitJob has been done with that value.
WaitJob will not be called from running jobs. It will be only be called from the original thread that
invoked Oodle. If you are running Oodle from a worker thread, ensure that that thread is allowed to wait
on other job threads.
See About Oodle Job Threading Plugins
Discussion
Combine a base and added path to make an absolute path, correctlyParameters
| base | first part of path
|
| add | second part of path
|
| into | result is written here
|
| intoSize | bytes available in into
|
Discussion
Combines two paths to make an absolute path. Some of the rules followed :
If add is absolute, into = add
If add begins with ":" or "/" , then into is the drive from base and the full path from add
If add does not begin with a path delim and base does not end with a path delim, one is inserted
(in particular : to do a normal path concatenation, add should NOT start with a path delim)
Relative path actions in add like ".." and "." are respected, but not in base.
The funny DOS-style per-drive current directory reference (eg. "c:blah") is respected at the beginning of add.
The string put in into always has only the preferred path delim for this OS (OODLEX_PATH_DELIM).
| OodleX async handle operations |
|
|
OodleXHandle is the ubiquitous weak reference to an Oodle Async operation.
About OodleLZ
OodleLZ consists of two major API groups : OodleAPI_LZ_Compressors for simple
synchronous memory to memory compression (like OodleLZ_Compress) (in the Oodle2 Core lib), and OodleXAPI_LZ_Async
(in the Oodle2 Ext lib)
for helpers that coordinate IO and multi-threaded invocation of the simple compressors
(like OodleXLZ_ReadAndDecompress_Wide_Async).
The Oodle LZ compressors are lossless generic data compressors. They offer world-beating decode speed, with
good encode speed and compression ratio. They are particularly
well suited to binary "structured data" as is typically found in games.
The most basic Oodle LZ APIs (OodleLZ_Compress and OodleLZ_Decompress) just compress and decompress from
memory to memory, one whole buffer at a time.
There are several compressors offered in Oodle. They are listed in the OodleLZ_Compressor enum. Each has slightly
different tradeoffs in compression ratio, decode speed, and encode speed. Some of them have other special purpose uses,
such as sliding windows or incremental encoding. (feel free to contact oodle@radgametools.com if you have an unusual use case
and need guidance)
For each choice of compressor, you can also vary the OodleLZ_CompressionLevel. The compression level just varies the encode speed;
slower encodes give higher compression ratios. Typically decode speed is unaffected; it's simply trying harder to pack the data
as small as possible in the encoder.
You can also dial decode speed vs compression ratio using the OodleLZ_CompressOptions:spaceSpeedTradeoffBytes parameter.
Kraken (OodleLZ_Compressor_Kraken) is the best compressor to try first. It offers superb decode speed with high compression. It's excellent for data
loading in game, distribution, and most general purpose uses.
Mermaid & Selkie are some of the fastest decompressors in the world. They provide less compression than Kraken but
super fast decodes. Selkie is faster than LZ4 and offers more compression.
Leviathan offers very high compression with still excellent decode speed, 10X faster than LZMA/7zip.
Leviathan should be your choice when you need maximum compression and don't mind being 10-50% slower than Kraken.
Hydra is a meta-compressor that selects Leviathan,Kraken, Mermaid or Selkie for you. See About OodleLZ Hydra
The older Oodle compressors are now deprecated and should not be used (see FAQ: What are the Oodle deprecated compressors ?).
Stick to the new sea monsters.
Go ahead and try them with the Oodle Examples
All OodleLZ compressors can optionally break their data into "seek chunks" if
OodleLZ_CompressOptions:seekChunkReset is set. The granularity is set in OodleLZ_CompressOptions:seekChunkLen , but must
be at least 256k (OODLELZ_BLOCK_LEN). You must enable this at compress time.
Doing seekChunkReset hurts compression ratio slightly, but makes it possible to seek in the packed data
and decompress just a portion without decompressing the whole thing. It also
allows parallel decompression. You may also want an OodleLZ_SeekTable , which records
the locations of the seek points; it should be created from the compressed buffer at encode time
via OodleLZ_CreateSeekTable.
This is needed if you want Oodle to parallelize decompression for you via OodleXLZ_ReadAndDecompress_Wide_Async.
(see FAQ: Which OodleLZ should I use? for a guide to choosing a compressor)
(see FAQ: What are the speeds and ratios of the OodleLZ compressors and levels? for a table of performance for the different choices,
or run example_lz_chart : Example that makes a chart of OodleLZ options)
The normal OodleLZ_Compress and OodleLZ_Decompress calls require a full buffer to encode
or decode (eg. not streaming or incremental), though you can do individual seek chunks or
OODLELZ_BLOCK_LEN at a time. There is also
a streaming decoder, which can decode incrementally (eg. OodleLZDecoder_Create).
The OodleLZ compressed data does not include any header. You must store
the raw size and compressed size yourself.
Because OodleLZ data is headerless, it can be concated at seek chunk boundaries.
You can compress any multiple of OODLELZ_BLOCK_LEN bytes separately and simply
concatenate the compressed buffers together. This is how you can compress a file
that's too big to fit in memory in one piece. You can also use this to do your own
multi-threaded compression if you don't want to use Oodle's OodleXLZ_Compress_Async.
This is demonstrated in example_lz : Example demonstrating LZ compression and decompression lz_test_9.
The OodleLZ compressors can create a CRC of each compressed chunk so that you can
verify its integrity before decompressing (you must set OodleLZ_CompressOptions:sendQuantumCRCs). This CRC is not checked in decode by default,
but can be enabled via the OodleLZ_CheckCRC option.
Kraken (and other new compressors) can now do parallel decodes without seek points.
See About OodleLZ ThreadPhased Decode for more.
How to choose a compressor and options
The best way to choose a compressor is just to try them. An easy way is to run example_lz_chart : Example that makes a chart of OodleLZ options on your data file to get
a sample of how the various Oodle LZ options perform.
Different compression algorithms have different tradeoffs in terms of speed, memory use, and compression ratio.
The OodleLZ compressors are intended to be at the sweet spot for in-game decompression on current game platforms.
Their speed/ratio tradeoff is designed to make loading compressed data and then decompressing it faster than just
loading decompressed data (in most cases; the exact balance depends on the data and how compressible it is).
Some notes on how to tweak the compressor to meet your goals :
First of all, OodleLZ_CompressionLevel controls how much work the compressor does to
optimize the stream. This affects compression time (NOTE : encoding time, not decoding time), but makes better streams. The
better streams are both smaller and often decompress faster. So if your goal is to have
the smallest stream or the fastest to decompress, in all cases you want to use the
highest level of OodleLZ_CompressionLevel that you can tolerate (usually Optimal2 is as high as you need to go).
Note the OodleLZ_CompressionLevel you pass to OodleLZ_CompressOptions_GetDefault does
not have to match the one you use for running the compression.
In general, always start with OodleLZ_Compressor_Kraken as your first choice.
1. If you want maximum compression :
Use OodleLZ_Compressor_Leviathan . Don't compress small buffers independently; instead
append them together and compress them as one chunk.
Do not set OodleLZ_CompressOptions:seekChunkReset (make it false).
Dial OodleLZ_CompressOptions:spaceSpeedTradeoffBytes down. Perhaps try 32. This trades off decode
speed for smaller compressed files.
2. If you want maximum decompress speed :
Use OodleLZ_Compressor_Selkie.
For threaded decodes, set OodleLZ_CompressOptions:seekChunkReset to true. This
ensures that decompression can proceed in parallel.
Make and transmit a OodleLZ_SeekTable table so that the parallel decoder can find its start points without scanning the data.
You can do this manually via OodleLZ_CreateSeekTable , or you can use an OOZ or OOP file to do this for you.
There are options in OodleLZ_CompressOptions which can adjust decode speed, such as minMatchLen, dictionarySize,
and spaceSpeedTradeOffBytes. Contact Oodle support for advice on these.
3. About manually setting up OodleLZ_CompressOptions :
Normally you should just use OodleLZ_CompressOptions_GetDefault , but if you want the last bit of speed or compression ratio,
you can sometimes find some more win by playing with individuals options. (this is particularly true with unusual data that
does not fit well with the default heuristics).
- seekChunkReset : makes chunks independent; hurts compression but allows parallel decompression, as well as seeking.
- seekChunkLen : sets the length of seek chunks. Generally you want this to be as large as possible,
while providing enough chunks to utilitize all worker threads. A good way to set it is to call OodleLZ_MakeSeekChunkLen
with a seek point count of 8 or so.
- spaceSpeedTradeoffBytes : controls the compressors decision about whether it should favor speed of decompression or
small size. It has units of bytes per time; roughly it's the number of bytes that must be saved in compression to
make a choice that hurts decode time by some fixed unit of time. The default is 256. For maximum speed of decompress you can set spaceSpeedTradeoffBytes
to a larger number (512). For maximum compression set spaceSpeedTradeoffBytes to a small number (32). In normal use
the values provided by OodleLZ_CompressOptions_GetDefault are appropriate.
- minMatchLen : can be used to raise the minimum match length of a compressor (you can't make it lower than
the compressor's default). Increasing to 6 or 8 (for example) can sometimes be good for compression or decode speed
on some types of data, such as image prediction residuals.
- dictionarySize : can be used to limit the maximum offset, which may be useful for decode speed on devices with
small caches or slow RAM, and can be used to prepare data for sliding window decoding.
If you specify OodleLZ_FuzzSafe_Yes then the output buffer will never be exceeded, even on corrupt data.
All the new compressors support Fuzz safe decoding, and it doesn't cost any decode speed. I recommend
always using OodleLZ_FuzzSafe_Yes
See also example_lz : Example demonstrating LZ compression and decompression , example_lz_simple : Example demonstrating very simple LZ memory->memory compression using only Oodle Core, example_lz_overlap : Example demonstrating parallel overlap with OodleLZ, example_lz_chart : Example that makes a chart of OodleLZ options
Discussion
Return the current OodleXFileOpsVTableDiscussion
Contains the default file ops function vtable that is used whenever no other vtable is provided.
This begins life equal to the vtable of OodleX_GetOSFileOps , but can be changed.
To mutate use OodleX_SetDefaultFileOps
Discussion
OODLEX_PATH_DELIM is either forward slash or back slash, whichever is preferred for the current OS.Discussion
Oodle IO functions are path-delim agnostic. You only need this if you are using non-Oodle OS IO functions.
Oodle for Linux is provided as a lib. There are separate libs for 32 and 64 bit Linux :
lib/liboo2corelinux.a
lib/liboo2extlinux.a
lib/liboo2corelinux64.a
lib/liboo2extlinux64.a
lib/liboo2corelinux.so.##
lib/liboo2extlinux.so.##
lib/liboo2corelinux64.so.##
lib/liboo2extlinux64.so.##
where ## is the Oodle API major version number.
Choose either 32 or 64 bit libs. You may use the static lib (.a) or shared library (.so).
You may use just oo2core for Oodle Core, or also use oo2ext for OodleX.
You exe should link against the Oodle shared library with its major version number
(eg. liboo2corelinux.so.8) to ensure a compatible library is used. You can make a symbolic link
from liboo2corelinux.so to liboo2corelinux.so.8 so that -loo2corelinux can be used.
The SO of OodleX also contains Core. If you use the OodleX SO do not use the Core SO !!
The 64-bit build of Oodle is faster, so should be used when possible.
The debug build of the Oodle lib is also provided. Generally the release build of Oodle should be linked
with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems.
oo2core requires libm and libc and libstdc++. oo2core requires glibc version 2.3.4
oo2ext has more system requirements.
OodleX on Linux requires these standard libs to be linked in :
-lm -lpthread -lrt -lstdc++
Oodle on Linux assumes SCHED_OTHER and doesn't try to do anything with thread priorities.
If you don't set a log file name yourself, OodleX for Linux tries to write a log file to "/var/log/oodle/" by default. If you don't want this, you
may disable logging or change the log location in OodleXInitOptions. On most systems, Oodle will fail to
create the "oodle" subdir in "var/log/" due to lack of permissions; you must use an administrator account to
create that dir for Oodle (and make that dir writeable).
If Oodle fails to write the log to the desired location, it will instead write it to "." (current directory).
enum OodleXHandleAutoDelete
{
OodleXHandleAutoDelete_No = 0,
OodleXHandleAutoDelete_Yes = 1,
OodleXHandleAutoDelete_Force32 = 0x40000000
};
Discussion
When you spawn an async task and get an OodleXHandle back to track the task, with a normal
OodleXHandleAutoDelete_No handle you have to ensure that the handle is deleted at some point
(typically by calling OodleX_Wait with OodleXHandleDeleteIfDone_Yes).Enumerants
Discussion
Alternative you can make the handle self-deleting by creating it with the OodleXHandleAutoDelete_Yes option.
In that case you can still inspect the handle status with OodleX_GetStatus and OodleX_Wait, but when the handle completes
and deletes itself, you will get OodleXStatus_Invalid. You cannot detect Done vs. Error cases with an OodleXHandleAutoDelete_Yes
handle.
enum OodleXLogCallbackRetRet
{
OodleXLogCallbackRetRet_Continue = 1,
OodleXLogCallbackRetRet_Terminate = 0,
OodleXLogCallbackRetRet_Force32 = 0x40000000
};
Discussion
Return value for OodleXLogCallbackRet Enumerants
| OodleLZDecoder_MakeValidCircularWindowSize |
|
|
Discussion
Get a valid "Window" size for an LZParameters
Discussion
NOTE: circular windows are deprecated as of 2.9.0
Most common usage is OodleLZDecoder_MakeValidCircularWindowSize(0) to get the minimum window size.
Only compressors which pass OodleLZ_Compressor_CanDecodeInCircularWindow can be decoded in a circular window.
WARNING : this is NOT the size to malloc the window! you need to call OodleLZ_GetDecodeBufferSize() and
pass in the window size to get the malloc size.
Discussion
Allocate an OodleXHandle to a simple data-less event Parameters
Return Value
Discussion
An "event" simply stores a transition from Pending -> Done/Error and can be used to wait on something you can trigger.
OO_SINTa OodleLZ_Decompress( const void * compBuf,
OO_SINTa compBufSize,
void * rawBuf,
OO_SINTa rawLen,
OodleLZ_FuzzSafe fuzzSafe OODEFAULT( OodleLZ_FuzzSafe_Yes ),
OodleLZ_CheckCRC checkCRC OODEFAULT( OodleLZ_CheckCRC_No ),
OodleLZ_Verbosity verbosity OODEFAULT( OodleLZ_Verbosity_None ),
void * decBufBase OODEFAULT( NULL ),
OO_SINTa decBufSize OODEFAULT( 0 ),
OodleDecompressCallback * fpCallback OODEFAULT( NULL ),
void * callbackUserData OODEFAULT( NULL ),
void * decoderMemory OODEFAULT( NULL ),
OO_SINTa decoderMemorySize OODEFAULT( 0 ),
OodleLZ_Decode_ThreadPhase threadPhase OODEFAULT( OodleLZ_Decode_Unthreaded ) );Discussion
Decompress a some data from memory to memory, synchronously.Parameters
| compBuf | pointer to compressed data
|
| compBufSize | number of compressed bytes available (must be greater or equal to the number consumed)
|
| rawBuf | pointer to output uncompressed data into
|
| rawLen | number of uncompressed bytes to output
|
| fuzzSafe | (optional) should the decode fail if it contains non-fuzz safe codecs?
|
| checkCRC | (optional) if data could be corrupted and you want to know about it, pass OodleLZ_CheckCRC_Yes
|
| verbosity | (optional) if not OodleLZ_Verbosity_None, logs some info
|
| decBufBase | (optional) if not NULL, provides preceding data to prime the dictionary; must be contiguous with rawBuf, the data between the pointers dictionaryBase and rawBuf is used as the preconditioning data. The exact same precondition must be passed to encoder and decoder. The decBufBase must be a reset point.
|
| decBufSize | (optional) size of decode buffer starting at decBufBase, if 0, rawLen is assumed
|
| fpCallback | (optional) OodleDecompressCallback to call incrementally as decode proceeds
|
| callbackUserData | (optional) passed as userData to fpCallback
|
| decoderMemory | (optional) pre-allocated memory for the Decoder, of size decoderMemorySize
|
| decoderMemorySize | (optional) size of the buffer at decoderMemory; must be at least OodleLZDecoder_MemorySizeNeeded bytes to be used
|
| threadPhase | (optional) for threaded decode; see About OodleLZ ThreadPhased Decode (default OodleLZ_Decode_Unthreaded)
|
Return Value
Discussion
Decodes data encoded with any OodleLZ_Compressor.
Note : rawLen must be the actual number of bytes to output, the same as the number that were encoded with the corresponding
OodleLZ_Compress size. You must store this somewhere in your own header and pass it in to this call. compBufSize does NOT
need to be the exact number of compressed bytes, is the number of bytes available in the buffer, it must be greater or equal to
the actual compressed length.
Note that the new compressors (Kraken,Mermaid,Selkie,BitKnit) are all fuzz safe and you can use OodleLZ_FuzzSafe_Yes
with them and no padding of the decode target buffer.
If checkCRC is OodleLZ_CheckCRC_Yes, then corrupt data will be detected and the decode aborted.
If checkCRC is OodleLZ_CheckCRC_No, then corruption might result in invalid data, but no detection of any error (garbage in, garbage out).
If corruption is possible, fuzzSafe is No and checkCRC is OodleLZ_CheckCRC_No, OodleLZ_GetDecodeBufferSize must be used to allocate
rawBuf large enough to prevent overrun.
OodleLZ_GetDecodeBufferSize should always be used to ensure rawBuf is large enough, even when corruption is not
possible (when fuzzSafe is No).
compBuf and rawBuf are allowed to overlap for "in place" decoding, but then rawBuf must be allocated to
the size given by OodleLZ_GetInPlaceDecodeBufferSize , and the compressed data must be at the end of that buffer.
An easy way to take the next step to parallel decoding is with OodleXLZ_Decompress_MakeSeekTable_Wide_Async (in the Oodle2 Ext lib)
NOTE : the return value is the total number of decompressed bytes output so far. If rawBuf is > decBufBase, that means
the initial inset of (rawBuf - decBufBase) is included! (eg. you won't just get rawLen)
If decBufBase is provided, the backup distance from rawBuf must be a multiple of OODLELZ_BLOCK_LEN
About fuzz safety:
OodleLZ_Decompress is guaranteed not to crash even if the data is corrupted when fuzzSafe is set to OodleLZ_FuzzSafe_Yes.
When fuzzSafe is Yes, the target buffer (rawBuf and rawLen) will never be overrun. Note that corrupted data might not
be detected (the return value might indicate success).
Fuzz Safe decodes will not crash on corrupt data. They may or may not return failure, and produce garbage output.
Fuzz safe decodes will not read out of bounds. They won't put data on the stack or previously in memory
into the output buffer.
Fuzz safe decodes will not output more than the uncompressed size. (eg. the output buffer does not need to
be padded like OodleLZ_GetDecodeBufferSize)
If you ask for a fuzz safe decode and the compressor doesn't satisfy OodleLZ_Compressor_CanDecodeFuzzSafe
then it will return failure.
The fuzzSafe argument should always be OodleLZ_FuzzSafe_Yes as of Oodle 2.9.0 ; older compressors did not
support fuzz safety but they now all do.
Use of OodleLZ_FuzzSafe_No is deprecated.
Discussion
Log the last error on a fileParameters
| file | the IOQFile to query
|
Return Value
| return | true if any error was logged
|
Discussion
Calls OodleXIOQ_GetLastError and OodleXIOQ_LogLastError
struct OodleXMallocVTable
{
void * m_context;
void * (OODLE_CALLBACK *m_pMalloc)( void * context, OO_SINTa bytes );
void * (OODLE_CALLBACK *m_pMallocAligned)( void * context, OO_SINTa bytes , OO_S32 alignment );
void (OODLE_CALLBACK *m_pFree)(void * context, void * ptr);
void (OODLE_CALLBACK *m_pFreeSized)(void * context, void * ptr, OO_SINTa bytes);
OO_S32 m_bigAlignment;
void * (OODLE_CALLBACK *m_pMallocBig)(void * context, OO_SINTa bytes);
void (OODLE_CALLBACK *m_pFreeBig) (void * context, void * ptr);
OO_BOOL (OODLE_CALLBACK *m_pValidatePointer)(void * context, void * ptr, OO_SINTa bytes);
};
Discussion
Function pointer table used to install the OodleX memory allocation functions
Members
Discussion
Use OodleXMalloc_InstallVTable to register a vtable as the one you want OodleX to use.
More commonly let OodleX_Init set one for you.
enum Oodle_UsageWarnings
{
Oodle_UsageWarnings_Enabled = 0,
Oodle_UsageWarnings_Disabled = 1,
Oodle_UsageWarnings_Force32 = 0x40000000
};
Discussion
Whether Oodle usage warnings are enable or disabled.
Discussion
allocate a table, then scan compressed LZ stream to fill the seek tableParameters
Return Value
| return | pointer to table if succeeded, null if failed
|
Discussion
Same as OodleLZ_FillSeekTable , but allocates the memory for you. Use OodleLZ_FreeSeekTable to free.
seekChunkLen must be a multiple of OODLELZ_BLOCK_LEN.
seekChunkLen must match what was in CompressOptions when the buffer was made, or any integer multiple thereof.
| OodleX_SetHandleAutoDelete |
|
|
Discussion
change handle lifetime management Parameters
| h | OodleXHandle weak reference
|
| autoDelete | if OodleXHandleAutoDelete_Yes, the handle deletes itself when not pending
|
Discussion
Handles that are OodleXHandleAutoDelete_No must be deleted or they will leak. The normal way to
delete them is by calling OodleX_Wait with OodleXHandleDeleteIfDone_Yes .
A handle that deletes itself when done will then report OodleXStatus_Invalid to queries, because it no longer exists.
If you change a handle to OodleXHandleAutoDelete_Yes and it is already done, this function
will delete it immediately, and the returned Status will not be OodleXStatus_Pending.
| Oodle2 Ext API Documentation |
|
|
Discussion
free a pointer allocated by OodleXMallocBig Parameters
| ptr | pointer to free (must not be NULL)
|
Discussion
You cannot call OodleXFree on a pointer allocated by OodleXMallocBig.
Uses the current OodleXMallocVTable ; this is an error if ptr was allocated from a different VTable.
Discussion
Initialize Oodle, without options structParameters
Return Value
Discussion
The debugSystems and threads options are just easy ways of getting pOptions filled out for common
use cases. For fine control of individual settings, you can always set the values in OodleXInitOptions yourself.
This is just a shortcut to OodleX_Init_GetDefaults then OodleX_Init
NOTE : do not use this if you want minimal linkage.
| OodleXLZ_Compress_Wait_GetResult |
|
|
Discussion
Wait, get result, and delete the handleParameters
Return Value
| return | OodleXStatus_Done for success
|
Discussion
| OodleLZ_GetSeekTableMemorySizeNeeded |
|
|
Discussion
Tells you the size in bytes to allocate the seekTable before calling OodleLZ_FillSeekTableParameters
Return Value
| return | size in bytes of memory needed for seek table
|
Discussion
If you wish to provide the memory for the seek table yourself, you may call this to get the required size,
allocate the memory, and then simply point a OodleLZ_SeekTable at your memory.
Then use OodleLZ_FillSeekTable to fill it out.
Do NOT use sizeof(OodleLZ_SeekTable) !
| OodleLZ_Compressor_CanDecodeThreadPhased |
|
|
Discussion
OodleLZ_Compressor properties helper.Discussion
Tells you if this compressor can be used with the OodleLZ_Decode_ThreadPhase.
See About OodleLZ ThreadPhased Decode
Discussion
void-void-star callback func pointer
takes void pointer, returns void
Discussion
The number of raw bytes per "seek chunk"
Seek chunks can be decompressed independently if OodleLZ_CompressOptions:seekChunkReset is set.
Async dispatchers and high level helpers for OodleLZ
| OODLEX_FILE_OPEN_NO_RESERVE_SIZE |
|
|
Discussion
Pass for reserveSize to OpenFile calls if you don't want it to reserve any space
enum OodleXError
{
OodleXError_Ok = 0,
OodleXError_InvalidHandle = 1,
OodleXError_FileNotFound = 2,
OodleXError_NoAccess = 3,
OodleXError_BadParameters = 4,
OodleXError_Corrupt = 5,
OodleXError_Alignment = 6,
OodleXError_Malloc = 7,
OodleXError_Compressor = 8,
OodleXError_UnexpectedEOF = 9,
OodleXError_PreviousAsyncFailed = 10,
OodleXError_Close = 11,
OodleXError_Unknown,
OodleXError_Count,
OodleXError_Force32 = 0x40000000
};
Discussion
oodle error enum to get a platform independent simple error code Enumerants
| OodleX_Init_GetDefaults_Threads |
|
|
enum OodleX_Init_GetDefaults_Threads
{
OodleX_Init_GetDefaults_Threads_No = 0,
OodleX_Init_GetDefaults_Threads_Yes = 1,
OodleX_Init_GetDefaults_Threads_Force32 = 0x40000000
};
Discussion
Should GetDefaults enable any threads?
| FAQ: Do new Oodle versions break data compatibility ? |
|
|
The short answer is NO : Oodle LZ compressed data is intended to be a lifetime supported bitstream format.
New versions of Oodle can read all data written by previous versions of Oodle.
If you have old compressed assets, and you want to know can I update to a new version of Oodle (to get a bug fix
or performance boost) and still load the old compressed assets - YES , always.
For details, continue reading :
Oodle will always be able to load old compressed data. That is, newer code will be forever backward
compatible with old compressed data.
Once in a rare while I rev a format to add new capabilities to the bitstream. When I do that, new compressed
data cannot be loaded by older code.
But - my intention is that when possible there is an option to stick with outputing the old format. This is exposed as
OodleConfigValues::m_OodleLZ_BackwardsCompatible_MajorVersion
(the "major" version is the one in the middle ; eg. 2.5.0 has a major version of 5)
For example Oodle 2.3.0 rev'ed the Kraken compressed format; this was the changelog :
Release 2.3.0 - July 14, 2016
- change : WARNING Kraken data made by 2.3.0 by default cannot be loaded by Oodle 2.2.0 ; set m_OodleLZ_BackwardsCompatible_MajorVersion if you need that.
Oodle can always load data made by previous versions, but the reverse is not necessarily true.
So if you have a version of Oodle >= 2.3.0 it will load data that's made by earlier versions.
By default Kraken data made by Oodle >= 2.3.0 cannot be loaded by earlier versions.
If you set OodleConfigValues::m_OodleLZ_BackwardsCompatible_MajorVersion to 2 , then the new modes will be disabled and the newer Oodle can make data compatible with earlier versions.
This could be useful if you have already shipped a game with an older version of Oodle, and you want to deliver new content but don't want to force an update of the decoder, but you do want to take a new version of the encoder (say for a bug fix or whatever), so you can continue to make content that works with the older decoders that customers have.
Obviously when I add new compressors, they will make data that can't be loaded by earlier versions. That is not explicitly called out or handled by the "BackwardsCompatible" system - it's up to you to choose compressors that are supported in the versions you have.
For example Mermaid & Selkie start in Oodle 2.3.0 , so if you use them then earlier versions won't load that data.
The only time I would ever completely break the format, requiring a full recompress of all content, would be if there was a major bug that I couldn't fix any other way. That's never happened and is unlikely, but I definitely advise you to always keep originals of all data just in case.
Special note about Hydra :
If you use Hydra, it can make use of any of the compressors available. When a new compressor is added to Hydra, it could make
compressed data that is not compatible with older versions. eg. in Oodle 2.6.0 Leviathan is added to Hydra, which means that Hydra
data made using Oodle 2.6.0 cannot be loaded with earlier versions of Oodle. This can be changed by setting m_OodleLZ_BackwardsCompatible_MajorVersion to 5,
which will exclude Leviathan from Hydra's menu to ensure it makes data that can be loaded with the previous version.
| t_fp_OodleCore_Plugin_Printf |
|
|
Discussion
Function pointer to Oodle Core printfParameters
| verboseLevel | verbosity of the message; 0-2 ; lower = more important
|
| file | C file that sent the message
|
| line | C line that sent the message
|
| fmt | vararg printf format string
|
Discussion
The logging function installed here must parse varargs like printf.
verboseLevel may be used to omit verbose messages.
| OodleLZ_CompressionLevel_GetName |
|
|
Discussion
Provides a string naming a OodleLZ_CompressionLevel compressSelect
Release 2.9.11 - Oct 9, 2023
- new : Oodle Data and Oodle Network for Apple VisionOS (beta).
- change : Xbox One legacy XDK support removed. Xbox One GDK/GameCore continues to be suppported.
- change : Xbox built with GDK March 2023 Update 6
- change : PS4 built with SDK 10.508
- change : PS5 built with SDK 7.00
- change : Switch built with SDK 16.2.3
- change : Data : Usage warning when decoding to slow-to-read (likely uncached) memory updated to provide more detail.
- enhancement : Texture : Improved end-to-end latency in multi-threaded encoding, especially for BC1 and BC4/5. (Activity tiling)
- enhancement : Texture : AVX-512 paths are significantly faster on AMD Zen 4 CPUs. (Avoid memory-destination forms of VPCOMPRESSD)
- enhancement : Texture : re-designed BC7 mode/partition selection logic that typically gives better quality while simultaneously encoding faster. Especially for diffuse albedo, we've seen RMS error reduced by 2-3% while simultaneously being about 2x faster to encode at "Normal" or "High" effort level.
- fix : Texture : BC7 RDO fix bug in "preserve extremes" mode where extremal alpha values weren't correctly preserved in certain rare cases. (Add check for mode 7 pbit mismatch in are_indices_permitted check)
- regression : PS4 Leviathan decode speed about 2-3% slower after compiler update
Release 2.9.10b - Apr 19, 2023
NOTE: re-release due to a buffer overflow issue in Oodle Texture 2.9.10 (Apr 17), please update immediately.
- new : Data : OodleLZ_GetCompressScratchMemBoundEx which can estimate typical, not just worst-case compression scratch memory bounds.
- enhancement : Data : Mermaid "Optimal1" (level 5) and higher levels now encode much faster (35-50% encode time reduction is typical), usually with slightly increased compression ratio as well.
- enhancement : Data : Selkie, Kraken and Leviathan "Optimal1" (level 5) levels have faster encoding (10-25% encode time reduction is typical). "Optimal2" and higher levels slightly faster as well.
- enhancement : Texture : Speed-ups to RDO activity mask generation, RDO encoding of all formats got slightly faster as a result.
- fix : Texture : RDO threading changes in 2.9.6 disabled threading of activity calc in most cases, re-tune heuristics.
- fix : Texture : Fix assertion failure when encoding 16384x16384 pixel BC5 textures from 4x U16 source data.
- fix : Texture : Fix float input pixels for non-BC6 formats triggering assertions when RDO encoding except for tiny input images.
- fix : Texture : Fix incorrect RDO encoding of BC6H surfaces with non-multiple-of-4 width or height and at least 16 blocks. Due to the interaction of these two criteria, typical power-of-2 textures with aspect ratios between 1:4 and 4:1 were generally unaffected. The bug typically results in higher visual fidelity and lower compression ratio than indicated by the target lambda, i.e. sub-optimal compression but no objectionable artifacts.
- fix : Texture : Input pixel format validation is now stricter. Previous versions erroneously accepted some unsupported input formats, e.g. 4x U16 pixel formats for BC1 encoding, then failed with internal errors later.
- note : Oodle 2.9.10 will be the last release with Xbox One XDK libraries. Xbox One support in upcoming releases will be GDK only.
Release 2.9.9 - Feb 8, 2023
This release focuses on Oodle Data compression speed.
- new : AArch64 (ARM 64-bit) simulator libraries for tvOS. tvOS distributions now also include the libs/headers packaged as an .xcframework.
- enhancement : Data : Kraken "Fast" (level 3) now takes typically around 10% less time.
- enhancement : Data : Kraken "Optimal1" (level 5) and higher levels encode in typically around 7-10% less time. This is independent of speed-ups from encoder scratch memory (see below) which can be in the same ballpark when encoding lots of independent chunks, sometimes much larger.
- enhancement : Data : Optimal1 and higher levels for all codecs now use provided scratch memory much more extensively and can sometimes avoid separate allocations completely if enough scratch memory is supplied. Especially when encoding many small, independent chunks, reusing allocations in this way can be a massive throughput increase on many-core machines. We've observed speed-ups of around 8.5x when encoding 256k Kraken chunks at Optimal2 level on a 64-core Threadripper. See "About Compression Scratch Memory" in the docs.
- enhancement : Data : OodleX parallel compress calls set up and reuse encoder scratch memory by default.
- enhancement : Data : "Optimal1" (level 5) compression on 64-bit ARM typically 10-15% faster independent of other changes noted above. (CTMF improvements)
- enhancement : Texture : BC7 RDO encoding is slightly faster, 5-8% reduction on typical textures in our test set.
- fix : Data : fix incorrect Linux 64-bit build settings for some files resulting in missing .note.GNU-stack sections and thus executable stacks.
- fix : Data : fix OodleX not balancing worker threads across multiple processor groups correctly if number of cores in a processor group wasn't 64. (Incorrect SetThreadGroupAffinity with ~0 affinity mask.)
- fix : Texture : Fix bug in float->sRGB conversion on x86 targets
Release 2.9.8 - Sep 28, 2022
- new : Oodle Data for WASM is now out of beta! Beta customers are encouraged to update due to smaller library size and encoder/decoder speed improvements. Please read "About Oodle on WASM" in the docs for integration notes.
- new : AArch64 (ARM 64-bit) simulator libraries for iOS. iOS distributions now also include the libs/headers packaged as an .xcframework.
- new : Texture : Support encoding from and decoding to float for BC1-5 and BC7 too, not just BC6H.
- new : Texture : BC1-3 and BC7 encode from float can now convert linear->sRGB (OodleTex_BCNFlag_LinearToSRGB), decode to float can convert sRGB->linear (OodleTex_BCNDecodeFlag_SRGBToLinear).
- new : Texture : Added OodleTex_DecodeBCN_Blocks_Ex and OodleTex_DecodeBCN_LinearSurfaces_Ex with "decode flags" argument to enable sRGB->linear encoding.
- change : iOS static libs now come as pairs of libs, libfoo.a and libfoo.sim.a, with the latter being for simulator. AArch64 (ARM64) targets exist for both device and sim so a single lib is no longer possible.
- enhancement : Data : Enable "destination is in write-combined memory" usage warnings for Windows/Linux/Mac x64 targets, not just game consoles
- enhancement : Texture : significantly improved rate-distortion performance of BC7 "preserve extremes" encoding
- enhancement : Texture : reduced worst-case memory use for BC1, BC3, BC4, BC5, BC7 RDO encoding (index merge heap size limit enforced).
- fix : Data : fix confusing symbol table entries around address 0 showing up in backtraces on NULL pointer fn calls on some ELF targets (NASM issue)
- fix : Data : fix memory leak in certain cases for Optimal2+ compression levels introduced in 2.9.6 (rrPool leak when allocation was attempted from scratch arena, but failed and fell back to user allocator instead)
- fix : Texture : Fix incorrect constant for "prefer wide vectors"/AVX-512 flag in public API (which didn't work correctly as a result)
- fix : Texture : Fix "preserve extremes" mode for BC7 RDO not preserving pixels correctly in certain rare cases (sometimes not round-tripping mode 7 endpoints in in-loop matrix pass)
- fix : Setting malloc/free allocator pointers to 0 through the plugin mechanism now installs always-failing dummy allocators instead of crashing on allocation.
Release 2.9.7 - June 30, 2022
- fix : Network : fix a crash bug in the dictionary builder introduced in 2.9.6 (m_scratch uninitialized)
- change : Data : OodleX Handle Table size is now growable up to 2048 times what it was before (2^m_num_handles_log2), making it practically unlimited
- change : Data : DirectStorage example updated for DirectStorage 1.0.2
- change : Texture : Encoder enables AVX-512 by default on more machine types.
- enhancement : Texture : Encoder now gives bitwise same results on all supported host platforms (Windows x64, Linux x64/ARM64, Mac x64/ARM64). This will be guaranteed going forward.
- enhancement : Texture : Significantly decreased memory usage for BC3-5 RDO encoding, slightly decreased for BC1-2.
- enhancement : Texture : "Preserve extremes" mode (preserves 0/1 in alpha channel exactly) is now supported for BC7. New BCNFlag name is OodleTex_BCNFlag_PreserveExtremes_BC3457, but the old OodleTex_BCNFlag_PreserveExtremes_BC345 will continue to be supported.
- enhancement : Texture : Encoding on ARM64 hosts is now significantly faster. On M1 Macs in particular, native ARM64 is now around ~2x faster than Rosetta x86_64 emulation, which used to be the faster path.
- note : Oodle 2.9.7 will be the last Oodle release to support 32-bit iOS, tvOS, MacOS targets or 32-bit x86 Linux. (Windows and Android 32-bit targets, as well as 32-bit ARM Linux, continue to be supported.)
Release 2.9.6 - May 2, 2022
- fix : Texture : fix an int overflow on textures 2 GB or larger, causing an effective under 2 GB limit for earlier versions, for example on RGBA float textures of 16k x 8k or larger.
- fix : Texture : fix incorrect assert in BC1 RDO encoder in debug builds when encoding BC1 textures without transparency that had fully black pixels
- fix : Texture : decreased error in baseline/RDO BC7 encodings for areas with certain singular covariance matrices
- new : Data : New example showing how to use Oodle Data with DirectStorage on Windows
- change : Data : improve error messages when trying to decode data that is not an Oodle stream
- change : Texture : Linux builds now enable AVX2 and higher support (previously disabled). Texture encoding on Linux should be significantly faster.
- change : Texture : Reduce number of small OodleJobs spawned, especially when compressing small mip levels.
- enhancement : Texture : BC1 RDO now produces slightly smaller results at same perceptual quality (or equivalently, better quality at the same size). The same improvement applies to RGB portion of BC2 and BC3.
- enhancement : Texture : AVX-512 support. We've seen up to 30% encode time reduction on AVX-512-supporting machines when encoding BC1, less for other formats. See "AVX-512" in the "Oodle Texture overviews" section in the documentation for usage notes.
Release 2.9.5 - October 13, 2021
- fix : Texture : fix bug in BC1 RDO encoder incorrectly emitting transparent black blocks in very rare cases
- fix : Texture : fix bug in BC1 RDO encoder not preserving BC1 transparency correctly in certain rare circumstances
- new : UWP ARM64 builds now provided
- change : Texture : BC1-3 decoders have changed, and the encoders now target the new decoders instead of the old ones (which matched the D3D reference). See "BCn decoding" in the "Oodle Texture overviews" section of the docs for rationale.
- enhancement : Texture : BC1-5 RDO encoding is significantly faster (often 3x or more); BC1 non-RDO encoding up to 2x faster; BC7 RDO encoding typically 1.13x faster.
Release 2.9.4 - September 2, 2021
This is a re-release of 2.9.3 with a change in compiler switches to fix a
binary compatibility issue on platforms compiled using MSVC. There are no
other changes.
- fix : Disable FH4 exception handling for link compatibility with VC 2017
Release 2.9.3 - July 26, 2021
- fix : Data : Fix crash encoding at level Optimal1 on buffers just over block size (eg. 256k+2 bytes) when matches cross block boundary (CTMF)
- fix : Data : Fix crash when calling OodleLZ_GetCompressScratchMemBound for OodleLZ_Compressor_None introduced in 2.9.1
- fix : Data : Fix debug lib on 32-bit Linux inadvertently using SSE4.1 in some static initializers prior to CPUID check
- fix : Data : Fix Oodle decompress erroneously reporting input as corrupted when the compressed data buffer is very close to the top of virtual memory
- new : Network : OodleNetwork1UDP_State_Compact_ForVersion and OodleNetwork1UDP_State_Uncompact_ForVersion allow you to read/write compacted Network state for old versions (eg. 5 for 2.5.x); compacted binary state changed from v5 to v6
- change : Linux built with Clang 12
- change : Linux built with Jcc erratum mitigation (-mbranches-within-32B-boundaries)
- change : NX built with SDK 11.4.3
- change : PS4 built with SDK 8.500
- enhancement : Faster Leviathan encoding at "Normal" and "Optimal" levels
- enhancement : CPU BC7Prep decode will now use AVX2 256-bit instructions when available, for a speed win of about 25% typically (it's frequently memory bound). You can disable usage of wide vectors when desired via OodleTexRT_BC7PrepDecodeFlags_AvoidWideVectors (see docs).
- regression : minor regression (5-6%) in encode speed for Mermaid/Selkie on PS4 at some compression levels after update to SDK 8.5 and clang 11
Release 2.9.2 - June 16, 2021
- fix : Texture : fix crash bug in RDO encoding with Universal Tiling option enabled on some texture sizes
Release 2.9.1 - June 7, 2021
- fix : Data : Fix incorrect warnings about OodleLZ_Compressor_None being deprecated
- fix : Texture : fix bug where large textures (eg. 8k x 8k) and high job worker counts (eg. 128) could cause an int overflow leading to a crash or huge alloc. Previous versions can work around this issue by clamping image size or worker count in OodleTex_Plugins_SetJobSystemAndCount.
- fix : Texture : fix bug in negative signed float values in BC6H RDO encoding producing garbage results.
- fix : Texture : fix high-error blocks in certain very rare cases when encoding BC1-3 textures without RDO at "High" setting
- change : MS platforms built with VC 2019 (except Durango-XDK still built with 2017)
- change : Windows built with JCC mitigation (/QIntel-jcc-erratum) and new FH4 exception handler
- change : Oodle Texture functions now have an explicit, enforced limit on surface dimensions (OODLETEX_MAX_SURFACE_DIMENSION)
- change : OodleTex_RMSE_Normalized_BCNAware on BC6/float changed metric, values will not compare to previous versions
- enhancement : Faster Kraken decompression on Zen
- enhancement : Faster decompression on Mac M1 ARM64
- enhancement : Faster Oodle Texture RDO encoding of BC4/BC5 even at highest quality level.
- enhancement : New entry point OodleTex_EncodeBCN_RDO_Ex that enables several new features like RDO effort levels and Universal Tiling.
- enhancement : Oodle Texure RDO encoding now has an "effort level" parameter, just like baseline encode. Lower effort levels are significantly faster to encode but reduce quality/compression slightly. Use for fast iteration, higher effort for final builds.
- enhancement : Oodle Texture Universal Tiling, to share a single RDO encode for linear and platform-specific tiled formats. Minor hit in compression rate, but much better than always RDO encoding in linear layout and much faster than encoding per-platform.
- enhancement : Can pass OODLETEX_JOBS_DISABLE to Oodle Texture functions to cause them not to spawn any jobs even when a job system is installed. Intended for cases that run many encodes in parallel anyway (e.g. Virtual Texture tiles) where spawning more fine-grained jobs isn't useful.
Release 2.9.0 - March 23, 2021
- fix : OodleLZ_Compress had an encoder crash bug in the Optimal level encodes on data in sizes just slightly over a 256KB chunk (eg. 262145) with a repeated substring at the very end
- change : Mac libs and dylibs are now fat binaries with x64 and ARM64
- change : Tex : Oodle Texture no longer checks for license file
- change : defining OODLE_IMPORT_LIB / OODLE_IMPORT_DLL is no longer needed; you can link with either type of lib without setting a define
- change : Oodle public headers no longer define types like U8, SINTa, they are instead OO_U8, OO_SINTa, etc.
- change : Oodle public headers now require stdint.h which on Windows/MSVC means VC2010 or higher
- change : Net : OODLE_PLATFORM_HAS_SELECTDICTIONARYANDTRAIN define removed. Call OodleNetwork1_SelectDictionarySupported instead.
- removed : Core : support for the deprecated LZ compressors is now removed (LZH,LZA,etc.). Only newlz (Kraken,Mermaid,Selkie,Leviathan,Hydra) and LZB16 are supported.
- removed : Core : OodleLZ_CompressContext_* functions removed; streaming encoders no longer supported
- removed : Ext : OODLEX_PATH_* public defines removed.
- removed : Ext : OODLEX_WCHAR_SIZE public define removed.
- removed : Tex : OodleTex_CheckLicense func removed ; Oodle Texture no longer checks for license files
- deprecation : OodleConfigValues::m_OodleLZ_Small_Buffer_LZ_Fallback_Size no longer used; newlz compressors no longer ever drop down to LZB16 (default behavior unchanged)
Release 2.8.14 - 2.8.x long term support release version
update March 21, 2021
- fix : OodleLZ_Compress had an encoder crash bug in the Optimal level encodes on data in sizes just slightly over a 256KB chunk (eg. 262145) with a repeated substring at the very end
- fix : PS4 decode speed regression fixed
Release 2.8.14 - February 15, 2021
- enhancement : BC7 encoding is faster ; slightly different encodings at higher speed with similar quality
- new : Mac ARM64 build now provided ; Mac example exes are fat x64+arm64
- new : Apple tvOS build now provided
- deprecation : Mac 32 bit x86 build no longer provided
- change : Xbox built with Nov 2020 GDK
- change : PS5 built with SDK 2.0
- change : PS4 built with SDK 8.0
- note : NX built with SDK 9.4.1
- note : Windows built with MSVC 2017
- regression : PS4 decode speed worse due to clang 10 in SDK 8.0
Release 2.8.13 - November 4, 2020
- fix : BC1 RDO could compute different results on Intel and AMD processors
- change : remove use of CRT in MSVC builds to remove -MT/-MD incompatibility
Release 2.8.12 - September 18, 2020
- change : Unreal Oodle Texture plugin now has support for all consoles
- change : Unreal Oodle Data plugin better support for soft disable and per-platform disable
- change : Texture full license files never stop working even if they are expired; eval licenses add 30 day grace period
- enhancement : Fix some regressions from 2.6.3 in OodleLZ encode & decode speed, particularly with Mermaid and Selkie on tiny buffers (tested on 1k and 4k)
- enhancement : clang10 Linux build with Jcc erratum mitigation workaround, and vc 2019, optional builds (gcc 47 and vc 2017 primary builds still available)
Release 2.8.11 - rerelease - April 11, 2022
- Call to rsqrt removed. Encoding will match previous behavior on Intel.
Use of reciprocal sqrt instruction caused texture encodings to be different on Intel and AMD processors.
Release 2.8.11 - August 20, 2020
- change : OodleTex_BC1_WithTransparency and OodleTex_BC2 now also support RDO ; OodleTex_BC1_WithTransparency can be used to preserve opaque alpha in the BC1 encoding.
- fix : Unreal integration for Oodle Texture was incorrectly using OodleTex_BC1 which has undefined alpha value; Unreal requires OodleTex_BC1_WithTransparency to output opaque alpha
- enhancement : OodleTex_BC1_WithTransparency non-RDO encode faster (was slower than BC1, now roughly same speed)
- enhancement : BC1-BC3 RDO quality very slightly better
Release 2.8.10 - August 4, 2020
- new : OodleTex_PixelFormat_GetName and OodleTex_BC_GetName helpers
- change : OodleTex_EncodeBCN_RDO and OodleTex_EncodeBCN_LinearSurfaces now accept more than 1 input surface with layout = NULL, so all mips/slices can be encoded in one call
- change : otexdds example now does all mips/volumes as a single Encode call for better parallelism
- change : default OodlePlugins_SetPrintf log plugin in MSVC platforms now uses stdio, creating a ucrt VC 2015+ dependency (previously all other platforms used stdio but MSVC platforms did not)
- change : (Windows only) when license is expired, it shows a message box and allows you to keep running rather than just failing
- fix : license file date compare could incorrectly report an expired license when it wasn't
- fix : potential race causing a crash in example_jobify_linuxtbb.inl (same as example_jobify_win32tp.inl fix in last release)
Release 2.8.9 - July 25, 2020
- new : Oodle Texture integration for UE4 (TextureFormatOodle for UE4.25.1)
- enhancement : Oodle Texture RDO encode much faster on large images
- change : Oodle Windows static lib build was using static lib CRT, changed to DLL CRT.
- change : OODLE_WORKERS_COUNT_ALL_HYPER_CORES can be used with OodleXInitOptions m_OodleInit_Workers_Count
- change : iOS libs incorrectly had the version number in the name (Oodle static libs do not have version number in the name, dynamic libs do)
- fix : Oodle Network training could fail on input larger than 2 GB (32 bit count overflow)
- fix : potential race causing a crash in example_jobify_win32tp.inl in dep link lifetime
Release 2.8.8 - July 7, 2020
- new : OodleTexRT_PS5GPU_BC7Prep_DecodeMulti to decode multiple BC7Prep chunks at once with reduced memory use and GPU synchronization overhead
- new : GPU-side BC7Prep decoding now supported on Xbox Series X, Xbox One and PS4
- enhancement : CPU-side BC7Prep decoding is significantly faster for large textures or when decoding to write-combined memory, and requires less scratch memory
- enhancement : PS5 GPU BC7Prep decode is 2x-3x faster, uses less GPU memory bandwidth, and significantly less scratch memory
- change : rename OodleTex_PixelFormat_3_F32_RGBA to OodleTex_PixelFormat_3_F32_RGB
- change : PS5 GPU BC7Prep no longer implicitly performs any cache invalidations; this is now up to the app.
- deprecation : OodleTexRT_BC7Prep_PS5GPU_Init is deprecated, call OodleTexRT_PS5GPU_Init instead
- deprecation : OodleTexRT_BC7Prep_PS5GPU_MinDecodeScratchSize is deprecated, call OodleTexRT_PS5GPU_BC7Prep_MinDecodeScratchSize instead
- deprecation : OodleTexRT_BC7Prep_PS5GPU_Decode is deprecated, use OodleTexRT_PS5GPU_BC7Prep_DecodeMulti instead
- fix : Xbox import lib names for DLLs had version number in the lib name; remove to match Windows names. Version number is in the DLL but not the import lib name.
- fix : 1 and 2 channel U16 promotion to 4 channel was not filling the implicit opaque alpha correctly, affected BC4 and BC5 decodes to 4 channel formats
Release 2.8.7 - June 8, 2020
- new : Oodle Texture! Oodle Texture is a new product separate from Oodle Data. Oodle Texture dramatically reduces the size of block-compressed BC1-BC7 textures.
- change : executables provided with SDK are now in bin dir
- change : oodle2base.h header with basic types now shared by Oodle Data, Net, and Texture
- change : example_jobify now provides a plug-in job system that can be used with Oodle Data, Net, and Texture
- change : t_fp_Oodle_Job common data type for Jobify functions in Oodle Data, Net, and Texture (replaces t_fp_OodleCore_Plugin_Job)
- change : OODLE_JOB_MAX_DEPENDENCIES common define for Jobify in Oodle Data, Net, and Texture (replaces OODLECORE_PLUGIN_JOB_MAX_DEPENDENCIES)
- change : _SALSA_ define no longer needed for PS5 Oodle header
- change : remove OodleLZ_CompressionLevel_Count ; the correct range is [Min,Max] because of negative levels
Release 2.8.6 - May 9, 2020
- new: OodlePlugins_SetJobSystemAndCount replaces OodlePlugins_SetJobSystem with desired parallelism argument
- enhancement : Leviathan decodes 5-10% faster on modern x86/64 platforms.
- change : Oodle build for Windows now compiled with MSVC 2017 and uses ucrt
- change : Oodle build for Mustard now compiled with MSVC 2017
- change : fix Mac and Linux debug & dead stripping
- deprecation : remove async Cancels from the public API (OodleX_CancelOrWait_AndDelete, OodleX_WaitCancelAllPending, OodleXIOQ_Pause, OodleXIOQ_WaitCancelAllPending)
- deprecation : m_OodleLZ_Desired_Parallel_BranchFactor removed from OodleXConfigValues, now automatically scaled to worker count
- deprecation : OodlePlugins_SetJobSystem is deprecated. Prefer OodlePlugins_SetJobSystemAndCount.
- deprecation : OodleXPriority levels are no longer exposed to the client. Client work should be OodleXPriority_Normal only. Prevents a potential deadlock bug mixing high priority client work with Core Jobs.
Release 2.8.5 - March 10, 2020
- fix : iOS build could assert "OodleAssert, 32 >= 48" in debug builds; this was also a bug in release builds on iOS that could cause stack variable corruption; do not use sendQuantumCRCs on iOS before this version!
- new: initial Mustard release version
- new: Xbox One GDK release version
- change : Stadia build variant
Release 2.8.4 - October 31, 2019
- fix : OodleNetwork1UDP_Encode could read one byte past the end of the input buffer, causing an access violation. The extra byte read was not used in coding, there's no problem with the compressed packets made. Allocating the input buffer with padding is an acceptable workaround for older versions.
Release 2.8.3 - October 4, 2019
- new: initial Salsa release version
- change : Stadia build SO extracts debug info to .debug file and uses add-gnu-debuglink to link to it
- change : disable debug info in Linux ARM64 build because of TLS debug relocation bug in clang
Release 2.8.2 - September 19, 2019
- fix : Rare non-critical issue : "WARNING: Too many indices for decoder scratch!" could be logged by the encoder in Leviathan Level 9. Valid compressed data was still made, but in some cases it was suboptimal.
- fix : Rare crash bug in Optimal level encoders on large buffers (LRM degeneracy jumpInShift 32) (since 2.8.1)
- fix : fix some spurious valgrind detections in Oodle (there were no bugs)
Release 2.8.1 - July 7, 2019
- new: Oodle Core API : OodleLZ_GetFirstChunkCompressor preferred name of old API OodleLZ_GetChunkCompressor
- new: Oodle Core API : OodleLZ_GetAllChunksCompressor should be used when chunks may have heterogeneous compressors (eg. with Hydra)
- new: Oodle Ext API : OodleX_CorePlugin_RunJob and OodleX_CorePlugin_WaitJob
- new: Oodle Ext API : OodleX_GetNumWorkerThreads
- enhancement : reduce OodleLZ_GetCompressScratchMemBound, particularly on small buffers and small hash table options, and more so for Mermaid/Selkie
- enhancement : OodleLZDecoder_MemorySizeNeeded was increased in version 2.8.0 to allow for apparent Mermaid/Selkie data possibly being Hydra ; it is now reduced back down and OodleLZ_GetAllChunksCompressor should be used to detect Hydra data.
- enhancement : Optimal level encoders 5-10% faster on large buffers (LRM Bloom filter)
- change : Hydra data now needs to be explicitly identified when used as the OodleLZ_Compressor type in functions like OodleLZDecoder_MemorySizeNeeded (or use OodleLZ_GetAllChunksCompressor)
- change : built with Switch SDK 7.4.0 (clang 7.0.1)
- change : built with PS4 SDK 6.5 (clang 7.0.1)
- fix : Switch : fix intermittent problem with debugging in the VSI in apps that use Oodle; Oodle linkage confused the debugger.
- fix : OodleLZ_GetCompressedStepForRawStep was incorrectly casting the return value to 32 bit, making it wrong for steps over 2 GB
- fix : fix rare read out of array bounds in Kraken & Leviathan encoder (multiarrays non-indexed splitter), would not be an access violation or compression failure, but could cause unexpected behavior and non-deterministic encoding.
- deprecation : OodleLZ_GetChunkCompressor will be removed in the future; use OodleLZ_GetAllChunksCompressor or OodleLZ_GetFirstChunkCompressor
- deprecation : OodleXMallocCall family of functions were incorrectly public, removed
Release 2.8.0 - April 9, 2019
- new : OodleCore_Plugins_SetJobSystem for Job plugin system to thread work in Oodle Core using a user-provided worker thread system (see Oodle_About_Job_Threading_Plugins)
- new : OodleLZ_GetCompressScratchMemBound to query how much scratch is needed for OodleLZ_Compress to avoid additional allocations
- new : example_lz_noallocs : example demonstrating Oodle compression & decompression with no allocations done by Oodle
- enhancement : Optimal level encoders are faster, particularly Kraken, and even more so with Jobify threading (but also faster single threaded)
- enhancement : Leviathan fast levels (SuperFast-Normal) encode much faster with a small decrease in compression ratio
- change : API : WARNING CompressOptions struct changed. Old fields have not moved, new fields have been added at the end of the struct. Zero initializing the new fields means "use default".
- change : CompressOptions:jobify option to control threading of the OodleLZ_Compress encoder in Oodle Core (see Oodle_About_Job_Threading_Plugins)
- change : CompressOptions:farMatchMinLen option to allow decode platform cache targeting of the encoded stream
- change : CompressOptions:spaceSpeedTradeoffBytes value of zero now means "use default" (256) to make it consistent with all other Options taking zero for "default". Use negative to get zero.
- change : docs are now html instead of chm
- change : Android x86_64 build
- change : add a Mermaid HyperFast4 level (previously HF4 was the same as HF3 in Mermaid)
- change : API : OodleLZ_GetDecodeBufferSize, OodleLZ_GetCompressedBufferSizeNeeded and OodleLZ_GetInPlaceDecodeBufferSize : take compressor argument to return smaller padding for the new codecs.
- change : API : OodleLZ_GetChunkCompressor : take compressed size argument to ensure it doesn't read past end
- change : Kraken Optimal1 level gets a bit less compression in trade off for much faster encoding
- fix : WARNING! Nasty destructive bug in the example code : examples calling "make_example_input" were stomping the input file. This was not in the Oodle libs, just the example code.
- fix : Leviathan no longer uses the allocator if sufficient scratch mem is passed in to OodleLZ_Compress
- fix : bug in Leviathan scratch accounting could rarely make it refuse to encode some chunks (valid compressed data would still be made, but it might be suboptimal)
- fix : Mermaid/Selkie in-place decompression could fail ("in place" decoding is when the input compressed and output decompressed buffers overlap)
- fix : OodleX worker threads restored to LIFO wakeup order after being incorrectly FIFO for a few versions (improves cache coherence)
- deprecation : OodleXLZ_ReadCompressWrite_Async API removed
- deprecation : CompressOptions previously deprecated maxHuffmansPerChunk now unused
- deprecation : CompressOptions verbosity deprecated; wasn't used by the new compressors anyway
Release 2.7.6 - December 18, 2018
- enhancement : Linux ARM64 : Mermaid/Selkie decode faster
- change : Android libs now in abi subdirectories
- change : added new usage warning that verifies the compressed buffer passed in to OodleLZ_Compress is at least OodleLZ_GetCompressedBufferSizeNeeded bytes long; can be disabled with Oodle_SetUsageWarnings
- fix : Fix Leviathan at Optimal levels using a large amount of stack memory (was 512k, now fits in 64k)
- fix : Fix the Unreal .build.cs rules for the Oodle integration on non-Windows platforms, for Unreal 4.20
- fix : Fix the Unreal data integration with bAsync compression turned on there could be a crash due to a race
- fix : Fix Oodle Network lib name on Switch ; was "oo2corenet" should be "oo2netswitch"
- fix : Fix a bug in the Mermaid/Selkie decode on Switch that could cause a fuzz safety failure (overrun on corrupt data)
Release 2.7.5 - October 31, 2018
- new : OodleLZ_Compressor_RespectsDictionarySize property query tells if a compressor obeys OodleLZ_CompressOptions:dictionarySize (all the new codecs do, some old ones do not)
- new : ozip -b (benchmark) command line argument, similar to zstd -b
- enhancement : Linux ARM64 : enable ASM optimized kernels in build
- change : clarify that seek chunk boundaries are relative to dictionary base, not current raw buffer
- change : old compressor LZH was doing archaic RLE mode at level 1 (superfast), now removed
- change : Oodle Windows SDK import libs back in the "lib" folder (temporarily was "import_lib")
- fix : Fix bug with seekChunkReset on very large buffers (over 1 GB)
- fix : OodleLZ_CompressOptions_Validate was not making seekChunkLen power of 2, which it must be
- fix : change OODLELZ_SEEKCHUNKLEN_MAX from 1 GB to half GB. (recommend using OodleLZ_MakeSeekChunkLen)
Release 2.7.4 - October 25, 2018
- fix : Fix bug in Leviathan Optimal5 (level 9) encoder reading uninitialized memory. The encoder did not crash and always made valid data, but it was not deterministic.
Release 2.7.3 - October 10, 2018
- enhancement : Mermaid, Kraken & Leviathan decompress faster in 32-bit by about 5% on x86, even more on ARM.
- change : Oodle windows SDK now ships DLLs and static libs. Libs are in "import_lib" and "static_lib". (dir names changed in 2.7.5). To use the static lib define OODLE_IMPORT_LIB.
- fix : Fix bug that could cause the Kraken & Leviathan encoder to crash at level 8 and 9.
Release 2.7.2 - September 10, 2018
- fix : Fix linkage in 32-bit iOS build
Release 2.7.1 SDK Update - August 13, 2018
- change : Switch SDK updated to 5.5
- change : PS4 SDK updated to 5.5
Release 2.7.1 - August 8, 2018
- fix : Fix a fuzz safety failure. Decoding corrupt or attack data with Mermaid or Selkie could cause a read access violation.
Release 2.7.0 - August 6, 2018
- change : Oodle Network is now a separate SDK and lib from Oodle Data compression (Core & Ext).
- new : OodleLZ_CompressOptions::profile option added (replaces "unused")
- new : example_lz / lz_test_13 shows how static dictionaries can be used with Oodle LZ via memcpy
- enhancement : Mermaid, Kraken & Leviathan decompress faster by 5-10%
- change : OodleHuffman public APIs removed
- change : OodlePlugins_ API set renamed to OodleCore_Plugins_ and OodleNet_Plugins_
eg. OodlePlugins_SetPrintf -> OodleCore_Plugins_SetPrintf
and OodleNet_Plugins_SetPrintf added
- change : PS3 and Xbox 360 builds removed
- change : Oodle Mac SDK now requires min OSX version 10.9 with libc++ instead of libstdc++
(note using the dylib prevents most of these problems; it's recommended to use the shared libs instead of the static libs if possible)
- deprecation : begin removal of archaic option; rename OodleLZ_CompressOptions::maxHuffmansPerChunk -> deprecated_maxHuffmansPerChunk
Release 2.6.3 Update 1 - July 3, 2018
- fix : iOS lib didn't include bitcode correctly
Release 2.6.3 - June 6, 2018
- new : new faster "HyperFast" compression levels; OodleLZ_CompressionLevel_HyperFast1-4 provide super fast encoding with lower compression ratios for real-time encoding needs. Available in Kraken, Mermaid & Selkie.
- new : example_lz_chart is now provided as a pre-built executable for evaluators on desktop platforms
- new : ozip executable provided on desktop platforms which acts like gzip; can be used for pipe or file compression.
- fix : Mermaid Normal encoder wasn't doing dictionary preload; this made compression ratio worse than it should be in parallel encodes (a tiny bit).
- fix : OodleXLZ_Compress_Async was running wide even when OodleXAsyncSelect_Wide flag was not set. Now uses only one thread if Wide is not set.
- fix : Compression of buffers larger than 1 GB, with Options enabled for seekChunkReset with seekChunkLen of 0.5 GB or less could create invalid compressed data which would fail to decode or not respect the requested seek chunking
Release 2.6.2 - April 30, 2018
- enhancement : improve parallelism of the OodleX Async wide compression
- fix : Some SIMD code could try to run the SSE4 variant even on CPU's that don't have SSE4, causing a crash. (Affects x86/x64 Windows, Linux & Mac)
- fix : XBox One libs failed BinScope due to NASM assembly objects setting wrong MASM version number
Release 2.6.1 - April 4, 2018
- enhancement : Huffman encoders slightly faster. Fast levels of Mermaid, Kraken, etc. about 1% faster to encode.
- fix : setting OodleLZ_CompressOptions::maxLocalDictionarySize above 2^26 could cause a crash in the optimal encoders due to S32 wrapping to negative
- fix : OodleLZ in backward compatible mode for version < 6 would not compress memset chunks correctly (would send them uncompressed); they did not fail to encode or decode, it was just inefficient
- fix : Selkie wasn't sending memset chunks; now does
- fix : Oodle Network could assert if only one packet was given to SelectDictionaryFromPackets. This bug affected the debug lib only.
- deprecation : Optimal encode levels of old codecs LZNIB, LZBLW and LZA removed. Optimal encodes with those codecs now use Normal level encoder.
Release 2.6.0 - Feb 27, 2018
- new : new compressor Leviathan !!
- new : new API Oodle_SetUsageWarnings ; usage warnings are on by default
- new : OodlePlugin_Printf_Verbose added; can be installed in Oodle Core via OodlePlugins_SetPrintf
- enhancement : Kraken & Mermaid optimal levels now get more compression (chunk adaptation & TLL parse)
- enhancement : Mermaid optimal levels now get more compression on some files (costing of entropy offsets)
- enhancement : Kraken & Mermaid generally achieve better space-speed with the >= 2.6.0 bit stream changes (smaller sizes and/or faster to decode)
- enhancement : Kraken, Mermaid & Selkie fast levels are now much faster to encode! (especially SuperFast and VeryFast)
- change : WARNING Kraken & Mermaid data made by 2.6.0 by default cannot be loaded by Oodle 2.5 and earlier ;
set m_OodleLZ_BackwardsCompatible_MajorVersion if you need that.
Oodle can always load data made by previous versions, but the reverse is not necessarily true.
- change : WARNING Oodle Network Compact/Uncompact is not compatible between before 2.6.0 and after 2.6.0
- change : OodleLZ_Small_Buffer_LZ_Fallback_Size default value changed to 0 so it never happens unless the client
modifies this value to something larger.
- change : example_lz : added lz_test_12 showing in place decoding
- change : Hydra can select Leviathan. Hydra-compressed data cannot be loaded by Oodle 2.5 and earlier because they
don't contain Leviathan. To use Hydra to make data compatible with previous versions, set m_OodleLZ_BackwardsCompatible_MajorVersion.
- change : OodleLZ_Compress API change , added scratch memory arguments so client can pass in pre-allocated memory to
eliminate internal Oodle alloc calls. (non-optimal levels only in the new compressor family; optimal levels & old compressors will still do allocations even if scratch is
provided)
- change : fix name of enums to match OodleX convention : OodleLog_VerboseLevel renamed to OodleXLog_VerboseLevel , OodleLog_StateFlags renamed to OodleXLog_StateFlags
- change : secret level 8 (OodleLZ_CompressionLevel_Optimal4) is now public ; no longer modifies compress options or changes space-speed tradeoff target
- change : OodleLZ_CompressionLevel_Optimal4 Mermaid & Kraken higher compression optimal parse
- change : Oodle for Switch is now built with SDK 3.4
- change : OodleHuffman_EncodeArrayU8 takes scratch memory argument to avoid internal allocation
- change : OodleXLZ_Decompress_ThreadPhased_Narrow_Async takes scratch memory argument to avoid internal allocation
- change : OodleLZ_Decompress API : default argument value of OodleLZ_FuzzSafe_No removed; recommend passing OodleLZ_FuzzSafe_Yes
- change : LZH and LZHLW removed from FuzzSafe set. If you were previously decoding them using FuzzSafe_Yes they will now fail.
- change : OodleXLZ_ReadAndDecompress_Stream_Async API : added decBufBase argument
- change : OodleLZ_CompressOptions_GetDefault at level 8 was returning a lower spaceSpeedTradeoffBytes ; don't do that anymore, the default is always 256 ; decode speed is mostly independent of compression level.
- fix : Mermaid/Selkie : some of the non-optimal encode levels were not respecting the dictionarySize option to limit the match window
- fix : Kraken/Mermaid/Selkie didn't support CompressionLevel_None correctly. They now pass through data uncompressed at that level.
- fix : Kraken/Mermaid/Selkie decoders had some potential fuzz safety violations (long matches at end of buffer)
- fix : OodleHuffman_DecodeArrayU8 wasn't doing CPU detection; it was slower than it should be if you didn't call other Oodle functions
- fix : the OodleXLZ_Compress_WriteOOZ family of functions took seekChunkLen as both as a function argument and via the OodleLZ_CompressOptions , creating a potential for mismatch ; the function argument is now removed.
- fix : Mermaid encoder was not putting backwards compatible streams correctly for major version 3 (2.3) compatibility (was always making streams for version >= 2.4.0)
- fix : Mermaid/Selkie encoder could overrun compressed buffer beyond OodleLZ_GetCompressedBufferSizeNeeded on some blocks. In parallel encoding mode this could cause corrupted compressed streams.
- fix : OodleXLZ_Decompress_ThreadPhased_Narrow_Async had a race in its finalization that could cause a read from freed memory
- fix : OodleLZDecoder_DecodeSome was incorrectly returning false when compAvail was too small to parse the first header (typically < 16 bytes); now returns true with no progress
- fix : OodleLZ_Decompress could return -1 for failure; it should always return 0 for failure (OODLELZ_FAILED); check <= 0 in previous versions
- warning : DEPRECATION : Oodle compressors that predate Kraken are being gradually deprecated. (eg. LZH, LZNA, etc.) You should always be encoding with compressors
from the new Kraken family (Kraken, Mermaid, Selkie, Leviathan, Hydra). Old decompressors will be supported for the forseeable future, so old data will still be loaded but
the encoders will be removed in some future version. Attempts to encode with outdated compressors is now logged as an Oodle Usage Warning. This can be disabled with Oodle_SetUsageWarnings.
The OodleLZ_Compressor enums for the old codecs are now hidden; you must define OODLE_ALLOW_DEPRECATED_COMPRESSORS before including oodle2.h to get them.
- obsolescence : EASTL is no longer used
- deprecation : old OodleLZ_CompressionLevel_RLE removed
- deprecation : OodleX OOZ functions removed (OodleXLZ_Compress_WriteOOZ_Async, etc)
- deprecation : OodleXLZ_Decompress_ThreadPhased_Wide_Async removed
Release 2.5.5 - August 22, 2017
- fix : A bug in the Kraken & Mermaid encoders was introduced in Oodle 2.5.0 ; any data encoded with Kraken or
Mermaid from Oodle 2.5.0 - 2.5.4 should be checked for correct decompression and re-encoded if needed. Those
versions may have made data which will fail to decode, or might claim to succeed decoding but produce bad bytes.
The cause was that huff streams in the encoder could collide and overwrite each other without detection.
This was most likely in whole-huff chunks at 128k boundaries, on trinary-aligned data (such as RGB BMP data).
This was more likely with m_OodleLZ_BackwardsCompatible_MajorVersion <= 2 but could happen in other cases.
Release 2.5.4 - August 8, 2017
- new : Windows UWP build added
- change : Oodle PS4 now built with SDK 4.500
- fix : omit-frame-pointer was set on too many builds; it is now only used in x86 (32-bit) builds
Release 2.5.3 - June 14, 2017
- fix : remove alloca use in rrHuffman that could cause crash on threads with small stack size on Linux
Release 2.5.2 - May 25, 2017
- fix : Kraken decoder had a bug that caused it to incorrectly return failure on valid compressed chunks
(due to the header corruption check not counting the scratch space needed with correct accounting of the thread phasing header)
- NOTE this change means OodleLZDecoder_MemorySizeNeeded is slightly larger now
- fix : fix handle leak in OodleXLZ_Compress_WriteOOZFile_AsyncAndWait when file open failed
Release 2.5.1 - May 13, 2017
- enhancement : Mermaid & Selkie encoding improved (faster and better ratio) on buffers <= 64k bytes long
- change : Oodle Mac OSX build now includes static libs and dynamic libs.
- change : Xbox One build is made with March 2017 XDK , still VC 2012 (but VC 2015 compatible)
- deprecation : LZB16 and LZBLW Optimal level encoders are gone. LZB with CompressionLevel >= Optimal1 now drops down to Normal. Use Selkie instead.
- obsolescence : libdivsufsort is no longer used
Release 2.5.0 - April 20, 2017
- new : OodleHuffman_EncodeArrayU8 and OodleHuffman_DecodeArrayU8 reintroduced as public APIs. (not compatible with old public Huffman API)
- enhancement : Kraken, Mermaid & Selkie - the non-optimal encode levels (SuperFast - Normal) get more compression and encode faster
- enhancement : Kraken, Mermaid & Selkie - faster to decode, especially on ARM (Android, iOS, Switch)
- enhancement : Hydra encoding now works in levels below Optimal
- change : NX now uses SDK 3.0
- fix : Kraken, Mermaid & Selkie - SuperFast encode level had bugs causing it to get much worse compression than necessary
- fix : LZB16 encoder at the Optimal1 level could crash due to uninitialized memory
Release 2.4.3 - February 14, 2017
- fix : Mermaid & Selkie encoders could cause an access violation reading past the end of the raw buffer if the
length was just over 64k mod 128k
- fix : Some compressors failed to decode files that were (128k+1)mod 256k bytes long due to bug in fuzz safety checks
- fix : Selkie encoder was incorrectly choosing offset compression in some cases, causing slower decodes
Release 2.4.2 - January 25, 2017
- new : Nintendo NX (Switch) support (SDK 0.17.13 and 1.0.0)
- new : example_lz_outputchunking : Example demonstrating cutting OodleLZ compressed output into fixed size chunks
- enhancement : Kraken, Mermaid & Selkie now support "in-place" decoding ; OodleLZ_Compressor_CanDecodeInPlace now
returns true for all OodleLZ decoders
- fix : a bug could occur in Oodle Ext 32-bit builds on systems with user memory in the top 2 GB
- fix : rename OodleLog_Printf macro to OodleXLog_Printf to match OodleX lib naming convention
- fix : LZB16 decoder crashed on 32-bit ARM Android. LZB16 was also used as the fallback for tiny buffers
in Kraken/Mermaid/Selkie/Hydra.
Release 2.4.1 - October 3, 2016
- new : OodleX_ReleaseThreadTLS function to release OodleX TLS resources in the rare usage that you churn
through a huge number of thread creates and destroys
- change : Clean up the behavior of OodleLZ_GetCompressedStepForRawStep when it is not given the entire compressed data;
now returns -1 for error and 0 for not enough data.
- fix : Fix race in the OodleThinSemaphore used in the example code via OodleX_Semaphore_Post; this was
only used by example_lz_threadphased , so should not affect any production code
- fix : Out of memory default handler would just infinite busy-loop on fgetc on non-Windows platforms
- fix : Android distribution didn't contain AArch64 libs
Release 2.4.0 - August 22, 2016
- new : Hydra ; automatically selects Kraken/Mermaid/Selkie
- new : example_lz lz_test_11 demonstrates sliding window with Kraken using memcpy
- new : Mermaid+ ; slightly higher compression, slightly lower speed ; control Mermaid vs Mermaid+ using spaceSpeedTradeoffBytes
- enhancement : LZNA level 8 (Optimal 3) tries several min match lengths
- fix : Selkie & Mermaid encoders could overrun the compressed buffer when data expanded
- fix : fix Mac dylib having a different name for its self-reference
- fix : fix Linux link error with gold on the Oodle static lib
- fix : fix iOS version min lowered to 7.0 (was 8.1)
- fix : fix race in some low level thread primitives caused by the threadprofiler. Showed up as ThreadPhased decodes hanging.
Release 2.3.0 - July 14, 2016
- new : Mermaid ! - Mermaid is a new super-fast-to-decode compressor with good compression ratios (comparable to ZLib)
- new : Selkie ! - Selkie is the fastest-decoding compressor, with low compression ratios (comparable to LZ4)
- new : new compressor property query OodleLZ_Compressor_CanDecodeFuzzSafe
- new : OodleConfigValues for core-only compressor config
- new : OodleConfigValues::m_OodleLZ_BackwardsCompatible_MajorVersion - set to 2 to make Kraken data that can be loaded by Oodle version 2.2.0
- new : lz_test_10 in example_lz demonstrates decoding quanta from a finite io buffer
- enhancement : Kraken is even faster to decode (10-20%) on all platforms
- enhancement : Kraken significantly faster to decode on ARM
- change : WARNING Kraken data made by 2.3.0 by default cannot be loaded by Oodle 2.2.0 ; set m_OodleLZ_BackwardsCompatible_MajorVersion if you need that.
Oodle can always load data made by previous versions, but the reverse is not necessarily true.
- change : OodleLZ Decompress functions args changed. New OodleLZ_FuzzSafe argument. Removed the allowed_compressor_mask.
- change : OodleLZ_CompressOptions dictionarySize option added to limit offsets
- change : OodleLZ_CompressOptions offsetShift option for old LZH encoder removed (now always 0)
- change : LZB16 decoder is now fuzz safe. Very slightly slower to decode.
- fix : LZB16 Optimal1 level could crash in encode of files > 2GB
- fix : OodleLZ_GetInPlaceDecodeBufferSize was too big on tiny buffers (it's now never bigger than complen + rawlen)
- fix : iOS is shipped as static libs with extension .a
Release 2.2.0 - May 11, 2016
- new : threaded Kraken decoder! See OodleLZ_About_ThreadPhasedDecode
- new : example_lz_threadphased contains a client-side implementation of a ThreadPhased decoder
- new : Kraken Optimal3 (Level 7) mode with higher compression ratio
- enhancement : Kraken optimized for ARM processors
- enhancement : Kraken optimal parse levels improved, about 1% more compression
- change : XBox1 now built with VS 2012, but Core lib is compatible with VS 2015 (Ext will work in VC 2012 but not in VC 2015)
- change : Windows SDK redist64 dir removed; 64-bit DLL's are just in "redist"
- change : examples now include oodle2.h with a relative path so they compile out of the box
- change : Oodle2 Core lib default plugins no longer use the CRT on Microsoft platforms. See OodleAPI_OodleCore_Plugins
- change : minor version bumped for API incompatibilities
- change : OodleLZ_Decode_ThreadPhase argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for 2-threaded Kraken decode. To retain previous behavior, add OodleLZ_Decode_Unthreaded to your call (or use default argument).
- change : allowed_compressor_mask argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for fuzz safety. To retain previous behavior, add OODLELZ_ALLOWED_COMPRESSOR_ALL to your call (or use default argument).
- removed : Oodle Huffman public APIs removed
Release 2.1.5 - April 26, 2016
- new : Kraken !! An amazing new compressor with high compression and fast decode speed!
- change : example_packet : remove some rarely used options
- change : PS4 library now built with SDK 3.0
- change : Xbox One library now built with March 2016 QFE1 , VS 2015
- fix : LZB16 and LZBLW decoders could over-read compressed data if the last quantum was memset
- fix : example_packet : better randomization of subsets and limits in TestOodleNetwork_SelectDictionaryAndTrain
- removed : OodleLZ_GetZipLikeCompressionSettings is gone
- deprecation : LZHLW and LZNIB and LZBLW compressors are now deprecated, use Kraken instead
Release 2.1.4 - March 20, 2016
- new : example_lz lz_test_9 demonstrating block composability rules
- enhancement : LZNA Normal parse compression improved
- enhancement : LZNA wasn't using SSE on Linux or Mac ; it now does, which makes those platforms much faster
- change : Oodle Logs at verbosity level 2 are now compiled out of release builds
- change : example_lz_chart compressor set changed
- change : OodlePlugins_SetAssertion can now be called with a NULL function pointer argument as an easy way to disable all assert handling
- change : example_lz test_8 shows how to use the WithContext calls for independent buffers
- change : OodleNetwork1UDP_State_Uncompact return bool to indicate success or failure on bad data
- fix : OodleLZ_CompressWithContext with NULL CompressOptions crashed
- fix : LZBLW Normal level encoder would crash when called with dictionary preload (or large buffers)
- fix : OodleNetwork1UDP fix fuzz safety for split large packets
- fix : OodleNetwork1UDP fix fuzz safety prevent reading past end of dictionary
- fix : OodleNetwork1UDP removed all used of release-asserts, now returns failure in those cases
- fix : OodleNetwork1UDP properly document OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN
Release 2.1.3 - Feb 1, 2016
- enhancement : OodleNetwork1UDP_Decode faster
- fix : OodleNetwork1UDP_Decode fuzz safety
- fix : OodleNetwork1UDP - fix possible bug in large split packet handling; now properly ensures that complen <= rawlen,
and complen == rawlen always means a pass-through uncompressed packet
- deprecation : OodleLZ_Compressor_CanSPUDecompress function removed
Release 2.1.2 - Jan 16, 2016
- new : BitKnit ! BitKnit is a new LZ compressor with high compression and good speed, between LZNA and LZHLW
- enhancement : Most LZ encoders are faster
- enhancement : LZNA Fast & Normal levels get more compression
Release 2.1.1 - Nov 25, 2015
- new : example_lz_chart make a neat text chart of the Oodle compressors & levels to test performance
- new : example_network_client simple Oodle Network client example
- fix : don't store OODLE_HEADER_VERSION in the example_packet data header; use OODLE2_VERSION_MAJOR instead
- enhancement : most decoders faster by around 10% !
- enhancement : LZNIB Normal & Optimal1 parsers much improved, more compression and faster decodes!
- enhancement : LZNIB decodes faster
- enhancement : LZNA about 10% faster to decode
- enhancement : LZHLW decoder about 10% faster in 64-bit
- enhancement : LZB encoder must faster in 32-bit
- enhancement : LZNA on iOS now uses NEON, almost 2X faster to decode!
- deprecation : LZH and LZA compressors are now deprecated; use LZHLW and LZNA instead
- PS4 library now built with SDK 2.5
- Windows library now built with MSVC 2013
- Mac & iOS libraries now built with XCode 7.1
Release 2.1.0 - Oct 10, 2015
- new : Oodle2 first Beta release
Release 1.45.1 - May 13, 2015
- fix : fix bug in the LZNA level 7 (Optimal3) encoder
Release 1.45.0 - May 11, 2015
- new : New LZNA compressor! More compression than LZA and fast to decode on modern CPUs.
Release 1.44.2 - March 4, 2015
- enhancement : LZA new optimal parse; faster at Optimal1 and more compression at Optimal3
- enhancement : LZNib Normal encoder faster, better optimal parse
- enhancement : LZB Fast & VeryFast encoder faster
Release 1.44.1 - December 9, 2014
- fix : example_packet state header was not the same in 32-bit and 64-bit builds
- enhancement : example_packet LZ compression level option added
Release 1.44.0 - November 16, 2014
- new : Mac sdk now includes 64 bit libraries
- new : Android and IOS sdk's available, for Oodle Network only
- new : Unreal Engine integration of Oodle Network is available
- change : example_packet has better trained model IO
Release 1.43.0 - September 8, 2014
- fix : LZA packet compressor introduced in 1.42 was getting much less compression than it should due a bug; fixed!
Release 1.42.0 - September 3, 2014
- change : reduce memory use of LZNib and LZBLW compressors at Optimal level
- new : LZA compressor now has sliding window encoding and decoding
- new : OodleLZ_CompressWithContext incremental API now supports LZA
- change : example_packet LZA option added
- change : Oodle is using the new "radtypes.h" shared header
- enhancement : OodleNetwork1 UDP new implementation - faster, a little more compression.
- fix : Optimal parse compressors could hang on very large degenerate files
Release 1.41.0 - July 10, 2014
- new : OodleNetwork dictionary selection by string matching; see OodleNetwork1_SelectDictionaryFromPackets and example_packet ; usually improves compression
- enhancement : OodleNetwork1 TCP memory use reduced to 84104 bytes per channel
- enhancement : OodleNetwork1 UDP is faster to encode ; saved states are incompatible with older versions
- enhancement : LZNIB VeryFast and Normal much faster to encode; see Oodle_FAQ_LZCompareTable for speeds
- enhancement : LZB16-Fast and VeryFast much faster to encode; see Oodle_FAQ_LZCompareTable for speeds
- change : rename OodleNetwork1 TCP variant to OodleNetwork1TCP for clarity
- change : better default hash table sizes for LZB16
- fix : Fix bug in which packets that expanded in OodleNetwork1 TCP could cause a write at compbuf[-1]
Release 1.40.0 - June 27, 2014
- change : OodleStaticLZP renamed to OodleNetwork1
- enhancement : LZHLW compressor significantly faster to encode at all levels, especially Normal
- new : New compressor LZA for very high compression!
- fix : example_packet had a hard-coded limit of 64k for packet sizes; that was removed
- fix : OodleLZDecoder_Reset was in the header but not implemented
- new : OodleNetwork1 compressed buffer size is now limited by OodleNetwork1_CompressedBufferSizeNeeded
Release 1.30.0 - April 14, 2014
- new : Minimal library initialization; see OodleX_Init_GetDefaults_Minimal , OodleX_Init_NoThreads , OodleX_Shutdown_NoThreads
- change : OodleX_Init no longer accepts NULL options
- change : Windows : OodleX_Init no longer does timeBeginPeriod(1) ; it's now left up to the app to decide if they want to do that
- change : api simplified; many unused APIs removed
Release 1.22.0 - March 31, 2014
- new : redo docs for Oodle Network packet compression
Release 1.21.2 - March 11, 2014
- fix : OOP tool failed to extract large packages due to exceeding threading limits
- fix : SPU LZ match offsets fixed again
- fix : ThreadProfiler code stripped completely in final build
Release 1.21.0 - March 1, 2014
- new : OodleNetwork1UDP_StateCompacted added to persist the trained state more compactly
- enhancement : OodleLZH and LZHLW decompressors further optimized; 5-10% faster
- enhancement : OodleLZH and LZHLW no longer does an allocation to prevent overrun
- enhancement : OodleNetwork1 for UDP networking significantly faster (about 300%)
- fix : SPU LZ match offsets could be too large by 1, causing corruption in rare cases
Release 1.20.0 - November 25, 2013
- new : Oodle now shipping "final" build without debug facilities
- new : OodleLZ_GetZipLikeCompressionSettings to make it easier to select compressor settings
- fix : fix possible mismatch between structs in the public header and structs in the lib
- fix : matchTableSizeLog2 wasn't being used correctly; made LZB and LZNIB compress less than they should
Release 1.19.0 - November 14, 2013
- new : OodleNetwork1 now provides a UDP (stateless) variant for unordered packet compression (beta)
- new : OodleHuffman_MultiHuffEncode for UDP unordered packet compression (beta)
- note : version 119 numbering adjusted to match its intention better. 1.major.minor
- fix : LZNib could fail when using "in-place" decompression
- fix : LZH and LZHLW could speculatively read past the end of compbuf in rare cases, potentially causing a (benign) access violation
- fix : LZ parallel compression with long-range-matcher could fail when parallel chunks didn't align to LRM chunks
Release 1.1.8 - November 4, 2013
- new : add OodleXLZ_Decompress_MakeSeekTable_Wide_Async , a simple parallel version of OodleLZ_Decompress
- new : add matchTableSizeLog2 to OodleLZ_CompressOptions ; controls encoder memory use and speed
- new : add Oodle_FAQ_LZCompareTable to make choosing a compressor and level easier
- enhancement : LZB16 encoder faster
- fix : LZ decoders could speculatively read past the end of compressed buffers; this could cause an
access violation if the memory after the compressed buffer was not readable. Fixed.
- fix : Fix issues with the WithContext family of functions when used with circular window encoding.
- fix : Fix issues with combining heterogenous OodleLZ data (with different seek chunk settings) causing the decoder
to make invalid parallel decodes
- fix : Fix issues with LZB when used with packed-raw-overlap decode mode
- enhancement : PS4 : OodleMalloc goes to "default" memory instead of heap memory for large allocations
Release 1.1.7 - October 23, 2013
- new : unified OodleLZ_CompressContext functions for incremental encoding, with and without sliding window
- new : LZB16 can encode & decode in a circular window
- fix : fix scheduling bug in OOZ IO->Decompression loader
- fix : Linux : fix LZB16 generation page faults due to movdqa gcc bug
- fix : don't lock Oodle threads to cores except on platforms where it is necessary
Release 1.1.6 - October 7, 2013
- new : new LZB compressor - LZ-Bytewise (and LZB-LargeWindow), the fastest of all, for when even LZNib isn't fast enough
- fix : OodleLZ_FillSeekTable could incorrectly mark heterogenous streams as being entirely independent seek chunks
- fix : Linux & Mac : Fix worker thread stack sizes too small, could crash OOP tool and other uses of the worker threads.
- fix : In-place decompression of incompressible data could fail if source overlapped dest.
- fix : Fix OodleLZ_Compress_Async not working (and related APIs, such as OodleLZ_Compress_WriteOOZ_Async) when
OodleAsyncSelect_None was passed for asyncSelect (now runs synchronously on the calling thread).
Release 1.1.5 - August 8, 2013
- enhancement : LZNib now has a sliding window encoder for streaming compression
- new : new OodleHuffman functions expose huffman-only coding
- new : OodleNetwork1 compressor, static dictionary ideal for MMO network compression (Beta)
Release 1.1.4 - May 6, 2013
- enhancement : large IO ops are now broken on the IO thread instead of at call time
- enhancement : large IO aps are now cancellable in progress and return correct status
- fix : OOP header could fail to read when it had a seek table
- enhancement : rev the OOZ header; add the size of the header
Release 1.1.3 - May 1, 2013
- fix : OOP header reader was truncating 64-bit file sizes to 32-bit
- new : lzadvanced_output_chunking in example_lz_advanced
- enhancement: OodleLZ_GetCompressedStepForRawStep allowed to seek quanta
Release 1.1.2 - April 28, 2013
- fix : Linux dir enumeration was including "."
- fix : Linux Open for WriteCreate was overwriting existing files too aggressively
- fix : Linux Open failure had debug perror left in
- change : better default log dirs and log names
- change : log file will open in "." if default log dir is not accessible
- change : Linux & Mac do log->prev cycle like Windows
- change : disable stack trace on .map file platforms for now
- enhancement: OodlePackage_FillWithFiles only decompresses chunks it needs (not all)
- enhancement: OodlePackage_SortFiles now uses extention file groups
Release 1.1.1 - April 24, 2013
- fix : Oodle_GetExtensionKey was not doing tolower as it should
- fix : OodlePackage_FillWithFiles could crash if files failed to load
- fix : fix memory overrun crash in DirListing
- fix : fix null deref in arg parse in OOP tool
- change: rename IOQ_DeleteFile_ to IOQ_Delete_ and make it work on dirs as well
- new: add OodleDirListing_DeleteDirContents
Release 1.1.0 - April 11, 2013
- new: LZNIB super fast to decode LZ variant added
- new: LosslessFilters for some common data transformations that improve compression
- change: Low level IO now allows aligned and unaligned IO modes
- enhancement: Compression seek chunk size now variable, set by client
- enhancement: All compressors can run encode and decode in parallel
- enhancement: Long Range Matcher for LZ on big buffers
- enhancement: (PS3) SPU waits now without thread switches
- enhancement: (PS3) SPU after_decompress replaceable function
Release 1.0.0 - August 24, 2012
Discussion
Function pointer for OodleXLog_SetCallbackParameters
Return Value
| return | whether to supress the message or not
|
Discussion
OodleXLogCallbackRet is provided by the client to take log messages.
It is called before other log outputs so that it has the chance to return OodleXLogCallbackRetRet_Terminate
and supress other output.
enum OodleX_Shutdown_LogLeaks
{
OodleX_Shutdown_LogLeaks_No = 0,
OodleX_Shutdown_LogLeaks_Yes = 1,
OodleX_Shutdown_LogLeaks_Force32 = 0x40000000
};
Discussion
bool enum
| t_fp_OodleNet_Plugin_DisplayAssertion |
|
|
Discussion
Function pointer to Oodle Core assert callbackParameters
| file | C file that triggered the assert
|
| line | C line that triggered the assert
|
| function | C function that triggered the assert (may be NULL)
|
| message | assert message
|
Return Value
| return | true to break execution at the assertion site, false to continue
|
Discussion
This callback is called by Oodle Core when it detects an assertion condition.
This will only happen in debug builds.
enum OodleXMalloc_OS_Options
{
OodleXMalloc_OS_Options_None = 0,
OodleXMalloc_OS_Options_GuardBig = 1,
OodleXMalloc_OS_Options_GuardBoth = 2,
OodleXMalloc_OS_Options_GuardFrees = 3,
OodleXMalloc_OS_Options_Count = 4,
OodleXMalloc_OS_Options_Force32 = 0x40000000
};
Discussion
Options enum for OodleXMalloc_GetVTable_OS
Enumerants
| OodleXHandleEvent_SetError |
|
|
Discussion
Set an OodleXHandleEvent to OodleXStatus_ErrorParameters
Discussion
The state transition from Pending->Error is one way. If the handle is OodleXHandleAutoDelete_Yes, it
goes away now.
Oodle for XBox One is provided as a lib.
lib/oo2core_xboxone.lib
lib/oo2ext_xboxone.lib
The debug build of the Oodle lib is also provided. Generally the release build of Oodle should be linked with all versions of your game (do not link the debug build of Oodle with the
debug build of your game typically). The debug build of Oodle is provided to help you track down problems.
since Oodle 2.8.5 :
Oodle for XBox One is built using MSVC 2017, and can be used with MSVC 2015-2019.
Oodle is provided for both XDK and GDK.
Oodle Core for XBox One does not use any stdio. The default log plugin outputs only to OutputDebugString.
Do not decompress directly into uncached graphics memory.
See FAQ: How do I decompress to graphics memory quickly?.
Oodle now automatically detects this and warns about it.
Discussion
OodleX_Semaphore_WaitDiscussion
NOTE : it is not intended that you use these in production. They are for use in the Oodle
examples. Replace with your own thread functions for shipping.
| OodleXIOQ_NameIsDir_AsyncAndWait |
|
|
Discussion
Convenience version of OodleXIOQ_GetInfoByName_AsyncAndWait
Discussion
Add a "fence" to the operation queueParameters
| fileRef | (optional) the file to associate the request with
|
| autoDelete | (optional) lifetime of the operation handle ; see OodleXHandleAutoDelete
|
| priority | (optional) priority of the operation ; see OodleXPriority
|
| dependencies | (optional) dependencies; the async op won't start until these are all complete; note : these are not freed, they must be autodelete or you must free them some other way.
|
| numDependencies | (optional) number of handles in deps array
|
Return Value
| return | handle to the operation, or 0 if it could not be started (usually due to invalid args)
|
Discussion
A fence is a NOP which can be used to schedule against other operations.
eg. if you have an OodleXIOQFile and want to block on any (unknown) operations on that file completing,
you can add a Fence op to the file and block on it; earlier requests will flush first, so when the fence is
done you know all previous requests are done.
Discussion
Opaque data type for OodleNetwork1TCP_StateDiscussion
This data is per-channel and adapts to the channel. There must be one for each
encoder and one for each decoder.
This data is initialized either with OodleNetwork1TCP_State_InitAsCopy or OodleNetwork1TCP_State_Reset.
You can allocate and free it yourself. It must be of size OodleNetwork1TCP_State_Size.
For compression only of server->client data, your server must have one of these objects for each transmission channel (client). The client must have a matching one to receive from the server. They must be kept in sync - each one must get the same calls to Encode or Decode in the same order. If they ever get out of sync (eg. due to lost connection), then they must both be reset in the same way. (either OodleNetwork1TCP_State_InitAsCopy or OodleNetwork1TCP_State_Reset)
| OodleLZ_GetInPlaceDecodeBufferSize |
|
|
Discussion
Get the size of buffer needed for "in place" decodeParameters
| compressor | compressor used; OodleLZ_Compressor_Invalid to make it enough for any compressor
|
| compLen | compressed data length
|
| rawLen | decompressed data length
|
Return Value
| return | size of buffer needed for "in place" decode ; slighly larger than rawLen
|
Discussion
To do an "in place" decode, allocate a buffer of this size (or larger). Read the compressed data into the end of
the buffer, and decompress to the front of the buffer. The size returned here guarantees that the writes to the
front of the buffer don't conflict with the reads from the end.
If compressor is one of the new codecs (Kraken,Mermaid,Selkie,Leviathan), the padding for in place decodes can be
very small indeed. It is assumed you will be passing FuzzSafe_Yes to the decompress call.
If compLen is unknown, you want an in place buffer size that can accomodate any compressed data, then
pass compLen = 0.
See OodleLZ_Decompress for more.